>Without those periodic full page images in the log, the storage layer would have to replay an infinitely long chain of small deltas to reconstruct a page for a read request. What was once a bounded O(checkpoint frequency) replay becomes an unbounded chain, leading to a spike in read latency and resource consumption.
I don't follow: read requests are not served from the WAL. They read the current state of the page from the buffer cache, where the page is updated after the change (FPI or not) is written to the WAL.
So, the general architecture described here is solid, and I support it, but I take issue with the "Lakebase" naming thing.
Disaggregated storage and disaggregated compute have been an open trend in DBMS development for the last half-decade. This is an obvious move with modern computing paradigms, and the academic literature has a standard name for it.
This feels like "JAMStack" from Netlify happening all over again.
I tweeted about this in 2022, as a general trend, and also from the RocksDB meetup emphasizing disaggregated storage:
This is essentially a re-explanation of Neon’s architecture as a blog post.
Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech.
The Postgres ecosystem is pretty unique in its ability to produce a “boring” stable product, innovate, stay F/OSS, and create financial outcomes for participants.
This a 100 times^. The Postgres ecosystem is remarkable and has managed to strike the balance between OSS and commercial successes in a way that most infra verticals have not.
13 comments
[ 2.5 ms ] story [ 27.6 ms ] threadI don't follow: read requests are not served from the WAL. They read the current state of the page from the buffer cache, where the page is updated after the change (FPI or not) is written to the WAL.
This appears to only have any effect with datalake style installs, where storage is separate from compute.
Not going to have any effect on those small postgres installs for that generic one off app.
[0] https://ducklake.select/
Disaggregated storage and disaggregated compute have been an open trend in DBMS development for the last half-decade. This is an obvious move with modern computing paradigms, and the academic literature has a standard name for it.
This feels like "JAMStack" from Netlify happening all over again.
I tweeted about this in 2022, as a general trend, and also from the RocksDB meetup emphasizing disaggregated storage:
- https://x.com/GavinRayDev/status/1607769112234823680
- https://x.com/GavinRayDev/status/1600666127025156096
[0] https://www.orioledb.com/
Amazing that the Postgres ecosystem got this software for “free” (as in at least a basic version of it is F/OSS, IIRC there wasn’t any core bits held back), and the extremely engineer-heavy company got to make money, AND they got bought out in true acquisition style by a larger player that truly benefits from the tech.
The Postgres ecosystem is pretty unique in its ability to produce a “boring” stable product, innovate, stay F/OSS, and create financial outcomes for participants.
Neon also only just disabled FPWs - so there is new substance here. We published a similar blog on Neon
https://neon.com/blog/turning-off-fpw-for-faster-writes
Many people just keep adding data and think "maybe it will be useful in future" till their system goes down.
Many of your data is essentially useless for anything in future.
You can simply have data retention policy and for most app this ensures your data does not grow top huge