If you understand Postgres's problems so well (though in reality, such problems can occur in many applications, and it's unclear how your rewritten version attempts to solve them), then why not address them in the main branch by becoming a contributor.
Less code and more value.
I think the issue isn’t a lack of want for fixing these issues, but the inertia and friction of getting them fixed.
The issues around the transaction ids and process per connection are well known, but the changes to the codebase to fix them would either constitute a backwards incompatible change that would change storage needs or an incredibly large rewrite of the codebase that breaks with decades of assumptions.
The json issue is a lot less of a problem as that’s net new. But some of these changes have been debated for years with no movement (and no lack of willing developers to tackle it) and at some point a fork or rewrite like this will happen. In my mind, all LLMs have done is made this work easier to do. If you have reservations about LLM’s doing this kind of work, no one is forcing you to use it, and I think it shows the utility of LLMs in that these kinds of things now can exist.
This article states multiple times that spinning up a process is expensive/very expensive. Is that really true? I ask out of ignorance.
“Compared to other ways of doing parallelism, processes are very expensive, both in terms of taking CPU resources but also the amount of time it takes to spin up a new process.”
“Because it’s expensive to spin up new processes, Postgres will only do this for long-running queries.”
Also:
“There’s been years of people talking about switching Postgres from a process model to a threading model, but nothing concrete has come out of that.”
I’ve read several times that on Linux, the cost between a process and a thread is relatively small.
It’s also important to distinguish between an os thread and a userspace thread. The author said “thread” without qualification, so I don’t know which he meant. Userspace threads can be many times lighter than processes.
Er, there are some odd claims with little to back them up.
It's hard to see how vacuum would cause an 'outage'. It does, as correctly stated in TFA, compete with the DB for i/o bandwidth and hence will slow things down a bit. Nothing which faster hardware can't solve.
The 32bit width of the transaction id hasn't been an issue in the past (with data stored on spinning rust, TPS were counted in the hundreds). Whether that is truly a problem with NVMe is to be demonstrated. But perhaps, just perhaps, if one generates billions of transactions in a day, PostgreSQL as we know it, isn't the right tool?
10 comments
[ 2.8 ms ] story [ 34.8 ms ] threadMaybe you should start from bug report without LLM-slop?
The issues around the transaction ids and process per connection are well known, but the changes to the codebase to fix them would either constitute a backwards incompatible change that would change storage needs or an incredibly large rewrite of the codebase that breaks with decades of assumptions.
The json issue is a lot less of a problem as that’s net new. But some of these changes have been debated for years with no movement (and no lack of willing developers to tackle it) and at some point a fork or rewrite like this will happen. In my mind, all LLMs have done is made this work easier to do. If you have reservations about LLM’s doing this kind of work, no one is forcing you to use it, and I think it shows the utility of LLMs in that these kinds of things now can exist.
“Compared to other ways of doing parallelism, processes are very expensive, both in terms of taking CPU resources but also the amount of time it takes to spin up a new process.”
“Because it’s expensive to spin up new processes, Postgres will only do this for long-running queries.”
Also:
“There’s been years of people talking about switching Postgres from a process model to a threading model, but nothing concrete has come out of that.”
I’ve read several times that on Linux, the cost between a process and a thread is relatively small.
It's hard to see how vacuum would cause an 'outage'. It does, as correctly stated in TFA, compete with the DB for i/o bandwidth and hence will slow things down a bit. Nothing which faster hardware can't solve.
The 32bit width of the transaction id hasn't been an issue in the past (with data stored on spinning rust, TPS were counted in the hundreds). Whether that is truly a problem with NVMe is to be demonstrated. But perhaps, just perhaps, if one generates billions of transactions in a day, PostgreSQL as we know it, isn't the right tool?