I'm honestly surprised it took them so long to reach this conclusion.
> That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way.
As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no real knowledge of the complexity of the code and wish the Postgres devs all the luck in the world to get this working smoothly.
Yeah. I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea. It's not like the globals were organized before! You're just making the ambient state (awful as it is) explicit.
Exactly, if you're now forced to put everything in one place you're forced to acknowledge and understand the complexity of your state, and might have incentives to simplify it.
I believe I can safely say that nobody acknowledges and understands the complexity of all state within that class, and that whatever incentives there may be to simplify it are not enough for that to actually happen.
Right but that would still be true if they were globals instead. Putting all the globals in a class doesn't make any difference to how much state you have.
> Putting all the globals in a class doesn't make any difference to how much state you have.
I didn't make any claims about the _amount_ of state. My claim was that “you're forced to acknowledge and understand the complexity of your state” (i.e., moving it all together in one place helps understanding the state) is plain-out wrong.
It's not wrong. Obviously putting it all in one place makes you consider just how much of it you have, rather than having it hidden away all over your code.
Yes, it’s the most pragmatic and it’s only “awful” because it makes the actual problem visible. And would likely encourage slowly refactoring code to handle its state in a more sane way, until you’re only left with the really gnarly stuff, which shouldn’t be too much anymore and you can put them in individual thread local storages.
> I think as a straightforward, easily correct transition from 2000 globals, a giant structure isn't an awful idea.
Agree.
> It's not like the globals were organized before!
Using a struct with 2000 fields loses some encapsulation.
When a global is defined in a ".c" file (and not exported via a ".h" file), it can only be accessed in that one ".c" file, sort of like a "private" field in a class.
Switching to a single struct would mean that all globals can be accessed by all code.
There's probably a way to define things that allows you to regain some encapsulation, though. For example, some spin on the opaque type pattern: https://stackoverflow.com/a/29121847/163832
A plain global can be accessed from other compiled units - agreed with no .h entry it is my=uch more error prone e.g. you don't know the type but the variables name is exposed to other objects
At most they'd be determined to be read only constants that are inlined during constant folding. This includes most integral sized / typed scalar values that fit into registers for the most part, and nothing you've taken the address of either - those remain as static data.
I think there might be a terminology mix-up here. In C, a global variable with the `static` keyword is is still mutable. So it typically can't be constant-folded/inlined.
The `static` modifier in that context just means that the symbol is not exported, so other ".c" files can't access it.
A static variable in C is mutable in the same sense that a local variable is, but since it's not visible outside the current compilation unit the optimizer is allowed to observe that it's never actually modified or published and constant fold it away.
Check out the generated assembly for this simple program, notice that kBase is folded even though it's not marked const: https://godbolt.org/z/h45vYo5x5
It is also possible for a link-time optimizer to observe that a non-static global variable is never modified and optimize that away too.
But the Postgres mailing list is talking about 2000 global variables being a hurdle to multi-threading. I doubt they just didn't realize that most of them can be optimized into constants.
Yea. Just about none of them could be optimized to constants because, uh, they're not constant. We're not perfect, but we do add const etc to TU level statics/globals that are actually read only. And if they are actually read only, we don't care about them in the context of threading anyway, since they wouldn't need any different behaviour anyway.
I think my bigger fear is around security. A process per connection keeps things pretty secure for that connection regardless of what the global variables are doing (somewhat hard to mess that up with no concurrency going on in a process).
Merge all that into one process with many threads and it becomes a nightmare problem to ensure some random addon didn't decide to change a global var mid processing (which causes wrong data to be read).
Access checking, yes, but the scope of memory corruption does increase unavoidably, given the main thing the pgsql-hackers investigating threads want: one virtual memory context when toggling between concurrent work.
Of course, there's a huge amount of shared space already, so a willful corruption can already do virtually anything. But, more is more.
We did this with a project I worked on. I came on after the code was mature.
While we didn't have 2000 globals, we did have a non-trivial amount, spread over about 300kLOC of C++.
We started by just stuffing them into a "context" struct, and every function that accessed a global thus needed to take a context instance as a new parameter. This was tedious but easy.
However the upside was that this highlighted poor architecture. Over time we refactored those bits and the main context struct shrunk significantly.
The result was better and more modular code, and overall well worth the effort in our case, in my opinion.
Also, even if a 2k-member structure is obnoxious, consider the alternative - having to think about and manage 2k global variables is probably even worse!
+significant and unknown set of new problems, including new bugs.
This reminds me of the time they lifted entire streets in Chicago by 14 feet to address new urban requirements. Chicago, we can safely assume, did not have the option of just starting a brand new city a few miles away.
The interesting question here is should a system design that works quite well upto a certain scale be abandoned in order to extend its market reach.
I think this is a situation where a message-passing Actor-based model would do well. Maybe pass variable updates to a single writer process/thread through channels or a queue.
Years ago I wrote an algorithmic trader in Python (and Cython for the hotspots) using Multiprocessing and I was able to get away with a lot using that approach. I had one process receiving websocket updates from the exchange, another process writing them to an order book that used a custom data structure, and multiple other processes reading from that data structure. Ran well enough that trade decisions could be made in a few thousand nanoseconds on an average EC2 instance. Not sure what their latency requirements are, though I imagine they may need to be faster.
Obviously mutexes are the bottleneck for them at this point, and while my idea might be a bit slower than a low-load situation, perhaps it would be faster when you start getting to higher load.
> I'm honestly surprised it took them so long to reach this conclusion.
I'm not. You can get a long way with conventional IPC, and OS processes provide a lot of value. For most PostgreSQL instances the TLB flush penalty is at least 3rd or 4th on the list of performance concerns, far below prevailing storage and network bottlenecks.
I share the concerns cited in this LWN story. Reworking this massive code base around multithreading carries a large amount of risk. PostgreSQL developers will have to level up substantially to pull it off.
A PostgreSQL endorsed "second-system" with the (likely impossible, but close enough that it wouldn't matter) goal of 100% client compatibility could be a better approach. Adopting a memory safe language would make this both tractable and attractive (to both developers and users.) The home truth is that any "new process model" effort would actually play out exactly this way, so why not be deliberate about it?
With regard to client compatibility there are related precedents for this already; the PostgreSQL wire protocol has emerged as a de facto standard. Cockroachdb and ClickHouse are two examples that come to mind.
From what I gather postgres isn't doing conventional IPC but instead it uses shared memory, which means the same mechanism threads use but with way higher complexity
IPC, to me, includes the conventional shared memory resources (memory segments, locks, semaphores, condition variable, etc.) used by these systems: resources acquired by processes for the purpose of communication with other processes.
I get it though. The most general concept of shared memory is not coupled to an OS "process." You made me question whether my concept of term IPC was valid, however. So what does one do when a question appears? Stop thinking immediately and consult a language model!
Q: Is shared memory considered a form of interprocess communication?
GPT-4: Yes, shared memory is indeed considered a form of interprocess communication (IPC). It's one of the several mechanisms provided by an operating system to allow processes to share and exchange data.
...
Why does citing ChatGPT make me feel so ugly inside?
I always understood IPC, "interprocess communication", in general sense, as anything and everything that can be used by processes to communicate with each other - of course with a narrowing provision that common use of the term refers to those means that are typically used for that purpose, are relatively efficient, and the process in question run on the same machine.
In that view, I always saw shared memory as IPC, in that it is a tool commonly used to exchange data between processes, but of course it is not strictly tied to any process in particular. This is similar to files, which if you squint are a form of IPC too, and are also not tied to any specific process.
> Why does citing ChatGPT make me feel so ugly inside?
That's probably because, in cases like this, it's not much different to stating it yourself, but is more noisy.
> Why does citing ChatGPT make me feel so ugly inside?
Its the modern let me Google that for you. Just like people don't care what the #1 result on Google is, they also don't care what ChatGPT has to say about it. If they did, they'd ask it themselves.
Postmaster would just share the already shared memory between processes (containing also the locks). That explicit part of memory would opt-in to thread -like sharing and thus get faster/less tlb switching and lower memory usage. While all the rest of the state would still be per-process and safe.
tl;dr super share the existing shared memory area with kernel patch
All operating systems not supporting it would keep working as is.
I've never really been limited by CPU when running postgres (few TB instances). The bottleneck is always IO. Do others have different experience? Plus there's elegance and a feeling of being in control when you know query is associated with specific process which you can deal with and monitor just like any other process.
But I'm very much clueless about internals, so this is a question rather than an opinion.
With modern SSDs that can push 1M IOPs+, you can get into a situation where I/O latency starts to become a problem, but in my experience, they far outpace what the CPU can do. Even the I/O stack can be optimized further in some of these cases, but often it comes with the trade off of shifting more work into the CPU.
It's not just CPU - memory usage is also higher. In particular, idle connections still consume signficant memory, and this is why PostgreSQL has so much lower connection limits than eg. MySQL. Pooling can help in some cases, but pooling also breaks some important PostgreSQL features (like prepared statements...) since poolers generally can't preserve session state. Other features (eg. notify) are just incompatible with pooling. And pooling cannot help with connections that are idle but inside a transaction.
That said, many of these things are solvable without a full switch to a threaded model (eg. by having pooling built-in and session-state-aware).
> solvable without a full switch to a threaded model (eg. by having pooling built-in and session-state-aware).
Yeeeeesssss, but solving that is solving the hardest part of switching to a threaded model. It requires the team to come terms with the global state and encapsulating session state in a non-global struct.
> That said, many of these things are solvable without a full switch to a threaded model (eg. by having pooling built-in and session-state-aware).
The thing is that that's a lot easier with threads. Much of the session state lives in process private memory (prepared statements etc), and it can't be statically sized ahead of time. If you move all that state into dynamically allocated shared memory, you've basically paid all the price for threading already, except you can't use any tooling for threads.
I see postgres become CPU bound regularly: Lots of hash joins, copy from or to CSV, index or materialized view rebuild. Postgis eats CPU. Tds_fdw tends to spend a lot of time doing charset conversion, more than actually networking to mssql.
I was surprised when starting with postgres. Then again, I have smaller databases (A few TB) and the cache hit ratio tends to be about 95%. Combine that with SSDs, and it becomes understandable.
Even so, I am wary of this change. Postgres is very reliable, and I have no problem throwing some extra hardware to it in return. But these people have proven they know what they are doing, so I'll go with their opinion.
Throw a ridiculous amount of RAM at it is more correct assessment. NVMe reads are still an “I/O” and that is slow. And for at least 10 years buying enough RAM to have all off the interesting parts of OLTP psql database either in shared_buffers or in the OS-level buffer cache is completely feasible.
It's orders of magnitude faster than SAS/SATA SSDs and you can throw 10 of them into 1U server. It's nowhere near "slow" and still easy enough to be CPU bottlenecked before you get IO bottlenecked.
But yes, pair of 1TB RAM servers gotta cost you less than half year's worth of developer salary
an array of modern SSDs can get to a similar bandwidth to RAM, albeit with significantly worse latency still. It's not that hard to push the bottleneck elsewhere in a lot of workloads. High performance fileservers, for example, need pretty beefy CPUs to keep up.
I've generally had buffer-cache hit rates in the 99.9% range, which ends up being minimal read I/O. (This is on AWS Aurora, where these bo disk cache and so shared_buffers is the primary cache, but an equivalent measure for vanilla postgres exists.)
In those scenarios,there's very little read I/O. CPU is the primary bottleneck. That's why we run up as many as 10 Aurora readers (autoscaled with traffic).
Postgres uses lots of cpu and memory if you have many connections and especially clients that come and go frequently. Pooling and bouncers help with that. That experience should better come out of the box, not by bolting on tools around it.
I don't get it. How is a 2000-member structure any different from having 2000 global variables? How is maintaining the struct possibly harder than maintaining the globals? Refactoring globals to struct members is semantically nearly identical, it may as well just be a mechanical, cosmetic change, while also giving the possibility to move to a threaded architecture.
Because global variables can be confined to individual cpp files, exclusively visible in that compilation unit. It makes them far easier to reason with than hoisting them to the "global and globally visible" option if you just use a gargantuan struct. Which is why a more invasive refactor might be required.
Yeah, I was really into that before there was even a cross-compiler/cross-platform syntax for declaring TLS values in C++ but have since “upgraded” to avoiding TLS altogether where possible. The quality of the implementations vary greatly from compiler and platform to compiler and platform, you run into weird issues with thread_at exit if they’re not primitive types, they run afoul of any fibers/coroutines/etc that have since become extremely prevalent, and a few other things.
What if the global variable has a greater scope than just a single TU? For simple variables of limited scope this approach would work but for more complex variables that are impacting multiple "modules" in the code it would introduce yet another code design problem to solve.
> I'm honestly surprised it took them so long to reach this conclusion.
Oracle also uses a process model on Linux. At some point (I think starting with 12.x), it can now be configured on Linux to use a threaded model, but the default is still a process-per-connection model.
Why does everybody think it's a bad thing in Postgres, but nobody thinks it's a bad thing in Oracle.
With the multi-threaded tcc above it scales about as well as multiprocess. With mainline it doesn't scale well at all.
So far I haven't gotten around to reusing anything across libtcc handles/instances, but would eventually like to share mmap()'d headers across instances, as well as cache include paths, and take invocation arguments through stdin one compilation unit per line.
I don't see the problem. All variables are either set in config or at runtime and then for every new query they are read and used by PostgreSQL (at least this is my understanding).
Regarding the threading issue, I think you can do the connections part multithreaded instead of one process per connection and still use IPC between this and postmaster. Because of the way PostgreSQL currently works, seems feasible to move parts one by one into a threaded model and instead of tens/hundreds of processes you can have just a few and a lot of threads.
Honestly, they should prototype it and see how it looks like and then decide on the way forward.
This would be one those places where a language like Rust would be helpful. In C/C++ with undefined behavior and crashes, process isolation makes a lot of sense to limit the blast radius. Rust borrow checker gives you at compile time a lot of the safety that you would rely on process isolation for.
Yes, but note that the blast radius of a PostgreSQL process crash is already "the whole system reboots", so there are not a lot of differences between process- and thread-based PostgreSQL written in C.
Rewriting in Rust would be interesting, but it would also probably be too invasive to make it worthwile at all - all code in PostgreSQL is C, while not all code in PostgreSQL interacts with the intrinsics of processes vs threads. Any rewrite to Rust would likely take several times more effort than a port to threads.
I feel this sort of undertaking could only be done by those programmers who truly value domain knowledge above all else (money, etc). I'm more of the entrepreneureal mind so I generally only learn as much as needed to do some task (even if it's very difficult), but just seeking information as a means to an end doesn't feel fulfilling to me. Of course many people DO find that, and its upon those people's shoulders that heroic things like this rest, and I'm very thankful to them.
I mentally snarked to myself that "obviously they should rewrite it in Rust first".
Then, after more thought, I'm not entirely sure that would be a bad approach. I say this not to advocate for actually rewriting it in Rust, but as a way of describing how difficult this is. I'm not actually sure rewriting the relevant bits of the system in Rust wouldn't be easier in the end, and obviously, that's really, really hard.
This is really hard transition.
I don't think multithread code quality should be measured in absolutes. There are things that are so difficult as to be effectively impossible, which is the lock-based approach that was dominant in the 90s, and convinced developers that it's just impossible difficult, but it's not multithreaded code that's impossibly difficult, it's lock-based multithreading. Other approaches range from doable to even not that hard once you learn the relevant techniques (Haskell's full immutability & Rust's borrow checker are both very solid), but of course even "not that hard" becomes a lot of bugs when scaled up to something like Postgres. But it's not like the current model is immune to that either.
This is true. However, the blast radius may be smaller with a process model. Also recovering from a fatal error in one session could possibly be easier. I say this as a 30-year threading proponent.
I'm assuming you're referring to formally proven programs. If that's the case, do you have any pointers?
Aside from the trivial while(!transactionSucceeded){retry()} loop, I have trouble proving the correctness of my programs when the number of threads is not small and finite.
It's not the same at all for global variables, of which pgsql apparently has around a couple thousand.
If every process is single threaded, you don't have to consider the possibility of race conditions when accessing any of those ~2000 global variables. And you can pretty much guarantee that little if any of the existing code was written with that possibility in mind.
Those global variables would be converted to thread locals and most of the code would be oblivious of the change. This is not the hard part of the change.
This sounds like a problem that would border on the complexity of replacing the GIL in Ruby or Python. The performance benefits are obvious but it seems like the correctness problems would be myriad and a constant source of (unpleasant) surprises.
The correctness problem should be handled by a suite of automated tests which PostgreSQL has. If all tests pass, the application must work correctly. The project is too big, and has too many developers to make much progress without full test coverage. Where else would up-to-date documentation regarding the correct behavior of PostgreSQL exist? In some developers head? SQLite is pretty famous for there extreme approach to testing including out of memory conditions, and other rare circumstances: https://www.sqlite.org/testing.html
Parallelism is often incredibly hard to write automated tests for, and this will most likely create parallelism issues that were not dreamed of by the authors of the test suite.
> If all tests pass, the application must work correctly.
These are "famous last words" in many contexts, but when talking about difficult-to-reproduce parallelism issues, I just don't think it's a particularly applicable viewpoint at all. No disrespect. :)
This is different because there isn’t a whole ecosystem of packages that depend on access to a thread unsafe C API. Getting the GIL out of core Python isn’t too challenging. Getting all of the packages that depend on Python’s C API working is.
An other component of the Gil story is that removing the Gil require adding fine grained locks, which (aside from making VM development more complicated) significantly increases lock traffic and thus runtime costs, which noticeably impacts single-threaded performance, which is of major import.
Postgres starts from a share-nothing architecture, it’s quite a bit easier to evaluate the addition of sharing.
> which (aside from making VM development more complicated) significantly increases lock traffic and thus runtime costs, which noticeably impacts single-threaded performance, which is of major import.
I don't think that's a fair characterization of the trade offs. Acquiring uncontended mutexes is basically free (and fairly side-effect free) so single-threaded performance will not be noticeably impacted.
Every large C project I'm aware of (read: kernels) that has publicly switched from coarse locks to fine-grained locks has considered it to be a huge win with little to no impact on single-threaded performance. You can even gain performance if you chop up objects or allocations into finer-grained blobs to fit your finer-grained locking strategy because it can play nicer with cache friendliness (accessing one bit of code doesn't kick the other bits of code out of the cache).
> which noticeably impacts single-threaded performance, which is of major import.
1) I don't buy this a priori. Almost everybody who removed a gigantic lock suddenly realizes that there was more contention than they thought and that atomizing it made performance improve.
2) Had Python bitten the bullet and removed the GIL back at Python 3.0, the performance would likely already be back to normal or better. You can't optimize hypothetically. Optimization on something like Python is an accumulation of lots of small wins.
You don’t have to buy anything, that’s been the result of every attempt so far and a big reason for their rejection. The latest effort only gained some traction because the backers also did optimisation work which compensated (and then was merged separately).
> Almost everybody who removed a gigantic lock
See that’s the issue with your response, you’re not actually reading the comment you’re replying to.
And the “almost” is a big tell.
> suddenly realizes that there was more contention than they thought and that atomizing it made performance improve.
There is no contention on the gil in single threaded workloads.
> Had Python bitten the bullet and removed the GIL back at Python 3.0
It would have taken several more years and been completely DOA.
Postgres already shares a lot of state between processes via shared memory. There's not a whole lot that would initially change from a concurrency perspective.
> there isn’t a whole ecosystem of packages that depend on access to a thread unsafe C API
They mentioned a similar issue for Postgres extensions, no?
> Haas, though, is not convinced that it would ever be possible to remove support for the process-based mode. Threads might not perform better for all use cases, or some important extensions may never gain support for running in threads.
Even the performance benefits are not big enough compare to the GIL.
Biggest problem of the process model might be the cost of having too many DB connections. Each client need a dedicated server process. Memory usage and the context switching overhead. Or if there is no connection pool, connection time overhead is very high.
This problem has been well addressed with a connection pool. Or having a middle ware instead of exposing the DB directly. That works very well so far.
Oracle has been supporting the thread based model and it's been usable for decades. I remember I tried the thread based configuration option (MTS or shared server) in 1990s. But no one likes that at least within my Oracle DBA network.
It would be a great research project but it would be a big problem if the community pushs this too early.
I've had a similar situation with PHP, where we had written quite a large engine (https://github.com/Qbix/Platform) with many features (https://qbix.com/features.pdf) . It took advantage of the fact that PHP isolated each script and gave it its own global variables, etc. In fact, much of the request handling did stuff like this:
It seemed so cool! PHP could behave like Node! It would have an event loop and everything. Fibers were basically PHP's version of Swoole's coroutines, etc. etc.
Then I realized... we would have to go through the entire code and redo how it all works. We'd also no longer benefit from PHP's process isolation. If one process crapped out or had a memory leak, it could take down everything else.
There's a reason PHP still runs 80% of all web servers in the world (https://kinsta.com/blog/is-php-dead/) ... and one of the biggest is that commodity servers can host terrible PHP code and it's mostly isolated in little processes that finish "quickly" before they can wreak havoc on other processes or on long-running stuff.
So now back to postgres. It's been praised for its rock-solid reliability and security. It's got so many features and the MVCC is very flexible. It seems to use a lot of global variables. They can spend their time on many other things, like making it byzantine-fault-tolerant, or something.
The clincher for me was when I learned that php-fpm (which spins up processes which sleep when waiting for I/O) is only 50% slower than all those fancy things above. Sure, PHP with Swoole can outperform even Node.js, and can handle twice as many requests. But we'd rather focus on soo many other things we need to do :)
I've been using PHP for decades and have found its isolated process model to be about the best around, certainly for any mainstream language. Also Symfony's Process component encapsulates most of the errata around process management in a cross-platform way:
Going from a working process implementation to async/threads with shared memory is pretty much always a mistake IMHO, especially if it's only done for performance reasons. Any speed gains will be eclipsed by endless whack-a-mole bug fixes, until the code devolves into something unrecognizable. Especially when there are other approaches similar to map-reduce and scatter-gather arrays where data is processed in a distributed fashion and then joined into a final representation through mechanisms like copy-on-write, which are supported by very few languages outside of PHP and the functional programming world.
The real problem here is the process spawning and context-switching overhead of all versions of Windows. I'd vote to scrap their process code in its entirety and write a new version based on atomic operations/lists/queues/buffers/rings with no locks and present an interface which emulates the previous poor behavior, then run it through something like a SAT solver to ensure that any errata that existing software depends on is still present. Then apps could opt to use the direct unix-style interface and skip the cruft, or refactor their code to use the new interface.
Apple did something similar to this when OS X was released, built on a mostly POSIX Darwin, NextSTEP, Mach and BSD Unix. I have no idea how many times Microsoft has rewritten their process model or if they've succeeded in getting performance on par with their competitors (unlikely).
Edit: I realized that the PHP philosophy may not make a lot of sense to people today. In the 90s, OS code was universally terrible, so for example the graphics libraries of Mac and Windows ran roughly 100 times slower than they should for various reasons, and developers wrote blitters to make it possible for games to run in real time. That was how I was introduced to programming. PHP encapsulated the lackluster OS calls in a cross-platform way, using existing keywords from popular languages to reduce the learning curve to maybe a day (unlike Perl/Ruby, which are weird in a way that can be fun but impractical to grok later). So it's best to think of PHP more like something like Unity, where the nonsense is abstracted and developers can get down to business. Even though it looks like Javascript with dollar signs on the variables. It's also more like the shell, where it tries to be as close as possible to bare-metal performance, even while restricted to the 100x interpreter slowdown of languages like Python. I find that PHP easily saturates the processor when doing things in a data-driven way by piping bytes around.
Is there any reason at all people use intrinsically bug-prone and broken multithreading mode instead of fork() and IPC apart from WinAPI having no proper fork?
TLB misses? They are just a detail of particular CPU implementation, and the architectures change. Also, aren't they per core and not per process? What would that solve then to switch to MT?
Worked on a codebase which was separate processes, each of which has a shedload of global variables. It was a nightmare working out what was going on, not helped by the fact that there was no naming convention for the globals, plus they were not declared in a single place. I believe their use was a performance move, ie having the linker pin a var to a specific memory location rather than copying it to the stack as a variable and referencing it by offset the whole time. Premature optimisation? Optimisation at all? Who knows, but there's a good reason coding standards typically militate against globals.
There's something to be said for globals whose access is well-managed, though.
IMO: if the variable is _truly_ global, i.e. code all over the codebase cares about it, then it should just be global instead of pretending like it's not with some fancy architecture.
The tricky part is reacting to changes to a global variable. Writing a bunch of "on update" logic leads to madness. The ideal solution is for there to be some sort of one-directional flow for updates, like when a React component tree is re-rendered... but that's very hard to build in an application that doesn't start out using a library like React in the first place.
Per discussion on this very page, in the headlined article, and in the mailing list discussion it references, PostgreSQL is not in that category. It has lots of static storage duration variables, which do not necessarily have external linkage.
Robert Haas pointed out in one message that an implementation pattern was to use things like file-scope static storage duration variables to provide session-local state for individual components. This is why they've been arguing against a single giant structure declared in "session.h" as an approach, as it requires every future addition to session state to touch the central core of the entire program.
They want to keep the advantage of the fact that these variables are in fact not global. They are local; and the problem is rather that they have static storage duration and are not per-thread, and thus are not per-session in a thread-per-session model.
There are 2000 globals here, so more like a couple of shedloads. While this is something you'd sort of expect for a product that's been around 30+ years, it really seems like there's a lot of optimization that could happen and still stick with the process model.
I recently looked through the source code of postgresql and every source files starts with a (really good) description of what the file is supposed to do, which made it really easy to get in to the code compared to other open source projects I've seen. So thanks for that.
I have no idea why that isn't standard practice in every codebase. I should be able to figure out your code without having to ask, or dig through issues or commit messages. Just tell me what it's for!
Because it takes a lot of time and because the comments can get outdated. I also want this for all my code bases. But do I always do this myself? No, especially on green field projects. I will sometimes go back and annotate them later.
Trying to understand what I previously wrote and why I wrote it takes more time than I ever care to spend. I'd much rather have the comments, plus at this point, by making them a "first class" part of my code, I find them much easier to write and I find the narrative style I use incredibly useful in laying out a new structure but also in refactoring old ones.
Even outdated comments can tell you the original purpose of the code, which helps if you're looking for a bug. Especially if you're looking for a bug.
If someone didn't take the time to update the comments and the reviewers didn't point it out, then you've probably found the bug because someone was cowboying some shitty code.
Outdated comments are often way worse than no comments, because they can give you wrong ideas that aren't true anymore, and send you off in the wrong direction before you finally figure out the comment was wrong.
It kind of is in rust now, with module-level documentation given its own specific AST representation instead of just being a comment at the top of the file (a file is a module).
On UNIX systems, Oracle uses a multi-process model, and you can see these:
$ ps -ef | grep smon
USER PID PPID STARTED TIME %CPU %MEM COMMAND
oracle 22131 1 Mar 28 3:09 0.0 4.0 ora_smon_yourdb
Windows forks processes about 100x slower than Linux, so Oracle runs threaded on that platform in one great big PID.
Sybase was the first major database that fully adopted threads from an architectural perspective, and Microsoft SQL Server has certainly retained and improved on that model.
> Windows forks processes about 100x slower than Linux...
I work with a Windows-based COTS webapp that uses Postgres w/o any connection pooling. It's nearly excruciating to use because it spins-up new Postgres processes for each page load. If not for the fact that the Postgres install is "turnkey" with the app I'd just move Postgres over to a Linux machine.
Could you point out, aside from the large numbers of clients I mentioned (and the development overhead of implementing multi-process memory management code), what the article mentions is a primary drawback of using processes over threads?
> The overhead of cross-process context switches is inherently higher than switching between threads in the same process - and my suspicion is that that overhead will continue to increase. Once you have a significant number of connections we end up spending a lot of time in TLB misses, and that's inherent to the process model, because you can't share the TLB across processes.
Yes, that's per-client performance scaling ("significant number of connections"), which indicates a pooled connection model might mitigate most of the performance impact while allowing some core code to remain process-oriented (and thus, not rewritten).
pgbouncer is not transparent, you loose features, particularly when using the pooling mode actually allowing a larger number of active concurrent connections. Solving those issues is a lot easier with threads than with processes.
That helps a lot but it's not a replacement for large number of persistent connections. If you had that you could simplify things in the application layer and do interesting things with the DB.
If you run postgres under WSLv1 (now available on Server Edition as well), the WSL subsystem handles processes and virtual memory in a way that has been specifically designed to optimize process initialization as compared to the traditional Win32 approach.
Didn't Oracle switch to threaded model in 12c - at least on Linux I remember there being a parameter to do that - it dropped the number of processes significantly.
I have. What I've observed more is outside attackers with their own agenda use the "nobody objected because they were unprepared and unable to respond in the 2 minutes I gave them to object" as proof their agenda is supported.
I'm curious if they can take advantage of vfork / CLONE_VM, to get the benefits of sharing memory and lower overhead context switches, with the trade of still getting benefits from the scheduler, and sysadmin-friendliness.
The other thing that might be interesting is FUTEX_SWAP / UMCG. Although it doesn't remove the overhead induced by context switches entirely (specifically, you would still deal with TLB misses), you can avoid dealing with things like speculative execution exploit mitigations.
Per the article, Postgres has many, many global variables, many of which track per-session state; much session state is “freed” via process exit rather than being explicitly cleaned up. Switching to CLONE_VM requires these problems to all be solved.
I wish they would do some kind of easy shared storage instead, or in addition too. This sounds like an odd solution, however I’ve scaled pgsql since 9 on very, very large machines and doing 1 pgsql cluster per physical socket ended up doing near-linear scaling even on 100+ total core machines with TB+ of memory.
The challenge with this setup is that you need to do 1 writer and multiple reader clusters so you end up doing localhost replication which is super weird. If that requirement was somehow removed that’d be awesome for scaling really huge clusters.
Pretty sure Tom Lane said this will be a disaster in that same pgsql-hackers thread. Not entirely sure what benefits the multi-threaded model will have when you can easily saturate the entire CPU with just 128 connections and a pooler. So I doubt there is consensus or even strong desire from the community to undertake this boil the ocean project.
On the other hand, having the ability to shut down and cleanup the entire memory space of a single connection by just disconnecting is really nice, especially if you have extensions that do interesting things.
> Tom Lane said: "I think this will be a disaster. There is far too much code that will get broken". He added later that the cost of this change would be "enormous", it would create "more than one security-grade bug", and that the benefits would not justify the cost.
>Not entirely sure what benefits the multi-threaded model will have when you can easily saturate the entire CPU with just 128 connections and a pooler.
That the all of those would work faster because of performance benefits, as mentioned in article
Heikki Linnakangas is one of the top Postgres contributors of all time, he isn't just "someone." The fact he's working for a startup on a fork (that already exists, which you can run right now on your local machine) doesn't warrant any snide dismissal. Robert Haas admitted that it would be a huge amount of work and that it would only be achievable by a small few people anyway, Heikki being among them.
Anyway, I think there are definitely limits that are starting to appear with Postgres in some spots. This is probably one of the most difficult possible solutions to some of those problems, but even if they don't switch to a fully threaded model, being more CPU efficient, better connection handling, etc will all go a substantial way. Doing some of the really hard work is better than none of it, probably.
For the record, I think this will be a disaster. There is far too much
code that will get broken, largely silently, and much of it is not
under our control.
regards, tom lane
If the existing code is old-school enough to use thousands of global variables in a thread-unsafe way, seems like changing it enough to compile as safe Rust code would push the "non-trivial" envelope pretty far.
There are several forks of PostgreSQL, in various levels of license, additional features and activity. However, maintaining a fork in addition to a main project is inherently more expensive than maintaining just a single project, so adding features to new major releases of the main project is generally preferred over forking every release into its own, newly named, project. After all, that is what we have major (feature) releases and stabalization windows (beta releases) for.
This should be considered a research effort, assuming it will be a complete rewrite. In light of that, you should not draw down resources from the established code base to work on it.
Ignoring the above, first state the explicit requirements driving this change and let people weigh in on those. This sounds like a geeky dev itch.
For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings.
Ten years of wasted time by the best internal PHP developers crippled the project - I'm amazed it survived at all.
I've used PHP in the past (PHP 4 and 5), as well as some simple templated projects in PHP 7. I try to keep up on news with what is happening in the PHP world, and it's difficult because of the hate for the language. Is the solution to Unicode strings still to just use the "mb_*" functions?
I got my real professional start using PHP, and have built even financial systems in the language (since ported to .NET 6 for my ease of maintenance, and better number handling). I'm still very interested in the language itself, in case I ever have the need to freelance or provide a solution to a client that can't afford what I can build in .NET (although to be honest, at this point I'm roughly able to code at the same speed in .NET as in PHP, but with the added type-safety, although I know PHP has really stepped up in providing this).
I believe so - most (all?) string functions have an mb_ equivalent, for working on multibyte strings.
Regular PHP strings are actually pretty great, since you can treat them like byte arrays. Fun fact: PHPs streaming API has an “in-memory” option and it’s… just a string under the hood.
Just don’t forget to use multibyte functions when you’re handling things like user input.
I have the "Professional PHP6" book which I feel like should be a collectors item or something.
Weird book IMO, because it has a lot of content that's just about general software development, rather than anything to do with PHP specifically, or the theoretical PHP6 APIs in particular.
PHP used to be the first computer language learned by people wanting to create a scripted web page. This was more true in the 90s but maybe it stuck. So it would be OK to add some general guidance about writing software and organizing projects.
I think this is a great article that takes a maximalist point and that’s its flaw.
You should rewrite code only when the cost of adding a new feature (one that is actually necessary) to the old codebase becomes comparable to designing your entire system from scratch to allow for that feature to be added easily. That is to say that the cost of the rewrite should become comparable to the cost of continuing development. I have been a part of a couple of rewrites like that, one of them quite complex, and yes they were warranted and yes they worked.
But having said that you should absolutely be conservative with rewriting code. It’s a bad habit to always jump to a rewrite.
I think it’s very dependent on how you use words like “rewrite” or “refactor”. The point the author makes about the two page function, and all the bug-fixes (lessons learned) makes sense only if you “rewrite” from scratch without looking at the history. You can absolutely “rewrite” the function in a manner that is “refactoring”, but will often get called “rewrite” in the real world. This may be because “refactor” is sort of this English CS term that doesn’t have a real translation or usage in many languages and “rewrite” is sort of universal for changing text, but in CS is sort of “rebuilding” things.
I don’t think you necessarily need to be conservative about rewriting things. We do it all the time in fact. We build something to get it out there and see the usage, and then we build it better and then we do it again. Which often involves a lot of “rewriting” but thanks to principles like SOLID’s single responsibility makes this rather easy to both do and maintain (we write a lot of semi-functional code and try to avoid using OOP unless necessary, so we don’t really use all the parts of SOLID religiously).
I do agree that it’s never a good idea to get into things with the mind-set of “we can do this better if we start from scratch” because you can’t.
There's currently a trend towards shitting on microservices-everything, imo largely justified. But missing from that is that identifying a logical feature and moving it to a microservice is one of the safer ways to begin a gradual rewrite of a critical system. And usually possible to get cross-dept buyin for various not always wholesome reasons. It may not always be the best technical solution but it's often the best political one when a rewrite is necessary.
>identifying a logical feature and moving it to a microservice is one of the safer ways to begin a gradual rewrite of a critical system
Why not identify that same logical feature and move it into a library. How does a Microservice add value here?
Identifying, extracting and adding tests to logical features has been the sane way to rewrite software for ages. Michael feathers even wrote a book about it [1]. This ship of theseus approach works because it's incremental and allows for a complete rewrite without ever having non-functional software in between.
Adding a REST/networking/orchestration boundary to make it a Microservice just for the sake of extracting it adds a lot of complexity for no gain.
Microservice can be the right architecture, but not if all you want is to extract a library.
Python3 changes were many "little" things; some more fundamental than other (unicode str). So I guess they were able to split the work in tiny pieces and, ultimately, were able to manage the project...
Yes, that was a rough migration process, but the long-term result is we have an improved language and growing community instead of Python going the way of PHP and Perl.
Between the 3 P's, Python's strategic decisions in 2000s were clearly the most successful.
The big problem there is that the people you are letting loose on the alternative, are lost from the original, so O loses steam that A gains. You still have to produce bug fixes and features to _both_ O and A to keep them in sync. So you essentially have a doubled required production rate to be delivered using the same staff.
So in order for there to be a net gain, the gang working on the alternative have to be able to find such big wins as to being neigh impossible.
This is a very very hard problem in our domain. 99% of the time, we have to simply resist the urge to _just rewrite the sucker_. No! Don't do it! (And this is incredibly hard because we all want to.)
I'm saying that the productivity gain would have to be incredibly large since it has to encompass a doubled output of features and bug fixes for a net zero change.
Say you have product A with features P, Q, R, and S; writing product B has to reproduce P, Q, R, and S, plus X and Y that is currently produced by the A team. On top of that, it has to fix (conceptual) bugs within P, Q, R, and S. All this is to be done by the new, crack, team that aims to make it so that: cost-of-development(B) < cost-of-development(A).
But the point is that the difference in magnitude of cost-of-development(B) and cost-of-development(A) has to be rather large considering the amount of work needed to have a return on that investment at all.
Process isolation affects so many things in C. The strategy change is going to require changes to so many modules that it will either be a re-write or buggy.
In practical terms, if every line needs to be audited and updated, it is a re-write
What makes you think that it will require that many changes? There will be some widespread mechanical changes (which can be verified to be complete with a bit of low level work, like a script using objdump/nm to look for non-TLS mutable variables) and some areas changing more heavily (e.g. connection establishment, crash detection, signal handling, minor details of the locking code). But large portions of the code won't need to change. Note that we/postgres already shares a lot of state across processes.
I'm not the person you asked and I don't have any particular knowledge of postgres internals.
Experience with other systems has taught me that in a system that's been in active use and development for decades, entanglement will be deep, subtle, and pervasive. If this isn't true of postgres then it's an absolute freak anomaly of a codebase. It is that in other ways, so it's possible.
But the article mentions there being thousands of global variables. And Tom Lane himself says he considers it untenable for exactly this reason. That's a very good reason to think that it will require that many changes imo.
A large refactor at best. It will touch lots of parts of the code base, but the vast majority of the source code would remain intact. Otherwise they could just Rewrite it in Rust™ while they’re at it
> if every line needs to be audited and updated, it is a re-write
I’m not sure why you believe every line needs to be updated. Most code is thread agnostic.
On the other hand, there's also the case of the Lunar Module guidance software that was hard-coded to run exactly every two seconds. If the previous subroutine call was still running when the next one was due, the previous one was harshly terminated (with weird side effects).
One of the main programmers suggested making it so that the next guidance routine wouldn't run until the previous one was done. This would make the code less sensitive to race conditions and allow more useful functionality for the pilots (who were the actual users and did seem to want it). However everyone assumed the two-second constant was implicitly embedded everywhere.
It wasn't -- only in a few places -- and with that fixed the code got more general and the proof of concept ran better than ever in about every simulator available. The amount of control it gave pilots was years ahead of the curve. But it never got a chance to fly on a real mission because what was there was "good enough" and nobody bothered to try.
In our combined comments there's a lesson about growing experiments and figuring out how to achieve failure quickly.
I don't expect you or others to buy into any particular code change at
this point, or to contribute time into it. Just to accept that it's a
worthwhile goal. If the implementation turns out to be a disaster, then
it won't be accepted, of course. But I'm optimistic.
The reply is much more reasonable than this blanket assertion of a disaster.
As an outsider it doesn't sound like something a few people could spin off in a branch in a couple months and see how code review goes. They're talking about doing it over multiple (yearly?) releases. It seems like it'll take a lot of expert attention, which won't be available for other work and the changes themselves will impact all other ongoing work.
I'm not trying to naysay it per se, bc again I don't have technical knowledge of this codebase. But that's exactly the sort of scenario that can cause a large project to splinter or stall for years. Talking about "the implementation" absent the context that would be necessary to create that implementation seems naively optimistic, or at worst irresponsible.
You are talking about implementation, the OP was talking about raising the concept with interested parties and seeing whether it is worth even starting to think about it.
They could fork, they could add threading to some sub systems and roll it out over several versions.
I don't know enough about the code but, of course, it is a hard problem but the solution might be to build it from the ground up as a threaded system, using the skills learned over 30 years and taking the hit on the rebuild instead of reworking what is there.
I am most interested because I didn't realise there was a performance problem in the first place.
Am I going crazy, or has the obvious implementation of such a change been missed on people? If they were proposing taking a multi-threaded app and splitting it into a multi-process one, I would predict they would find a hell of a lot of unexpected or unknown implicit communication between threads, which would be a nightmare to untangle.
Going the other way, there is an extremely well understood interface between all the processes which run in isolation: shared memory. Nearly by definition this must be well coordinated between the processes.
So the first step in moving to a multi-threaded implementation would be to change nearly nothing about each process, and then just run each process in its own pthread, keeping all the shared memory ‘n all.
You would expect performance to be about the same, maybe a little better with the reduces TLB churn, but the architecture is basically unchanged. At that point, you can start to look at what are more appropriate communication/synchronisation mechanisms now you’re working in the same address space.
I just don’t understand why so many people seem to think this requires an enormous rewrite - having developed as a multi-process system means you’ve had to make so much of the problematic things explicit and control for them, and none of these threads would know anything at all about each other’s internals.
Without being familiar with the Postgres source, this seems to be what I call a "somersault problem": hard to break down into sub-goals. I have heard that the Postgres codebase is solid which makes it easier but it's still mature and highly complex. It doesn't sound feasible to me.
The original post does describe several sub-problems. The group could first chip away at global state, signals, libraries. They can do this before changing the process model in any way.
Feel like the PostgreSQL Core Team should just build a new database from scratch using what they have learned from experience instead of attempting such a fundamental architectural migration. It would give them more freedom to change things also. Call it "postgendb" and provide a data migrator.
That's a great idea. I've been considering whether or not to use Cockroach Db at work, and I love the fact that it's distributed from the get go.
Why not work on something like that instead of changing something that works? Especially since they the process model really only runs into trouble on large systems.
Please don't use mutable global state in your work. Global variables are universally bad and don't provide much of a benefit. The number of desirable architectural refactoring that I've witnessed turning into a muddy mess because of them is daunting. This is one more example of this.
Thank you for sharing your ideological views, but this is not the appropriate venue for that. If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that. All engineering is trade offs. As soon as you make blanket statements that X is always bad, you’ve transitioned into the realm of ideology. Now presumably you mean to say it’s almost always bad. But that really depends on the context. It may well be almost always bad in average software projects, but PostgreSQL is not your average software project. Databases are a different realm.
Global mutable state being a poor choice in software architecture isn’t an ideology. There is no ideology that argues it is awesome.
If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that.
All engineering is trade offs. As soon as you start telling people they’re making blanket statements that X is always bad, you’ve transitioned into the realm of nitpicking.
Using globals is simpler, it's also pretty natural in event driven architectures. Passing everything via function arguments is welcome for library code, but there's little point to using it in application code. It just complicates things.
Knizhnik made these variables thread local, which is fine if you have a fixed association of threads to data. This looses some flexibility if your runtime needs to incorporate multiple sessions on one thread (for example to hide IO latency) in the future. In the end, the best solution is to associate the data that belongs to a session with the session itself, making it independent on which thread it's running on. This is described by Knizhnik as "cumbersome", which is exactly why people should have not started with global variables in the first place. (No blame, Postgres is from 1986 and times were very different back then).
It's awesome where performance considerations are paramount. It's awesome in databases. It's awesome in embedded software. It's awesome in operating system kernels.
The fact is sometimes it's good. Saying it's universally bad is going beyond the realm of logic and evidence and into the realm of ideology.
Discrediting my argument by labeling it as ideology and by implying that "blanket statements are always bad" is a logical fallacy that does not touch the merits of what is discussed and I would argue that your argument instead of mine is the one that does not belong here.
If you want to contribute to the discussion, I'd be happy to be given an example of successful usage of global variables that made a project a long term success under changing requirements compared to the alternatives.
You know what a database is, do you? It is the place where you store your mutable global state. You can't kick the can down the road forever, someone has to tackle the complexity of managing state.
376 comments
[ 3.3 ms ] story [ 324 ms ] thread> That idea quickly loses its appeal, though, when one considers trying to create and maintain a 2,000-member structure, so the project is unlikely to go this way.
As repulsive as this might sound at first, I've seen structures of hundreds of fields work fine if the hierarchy inside them is well organized and they're not just flat. Still, I have no real knowledge of the complexity of the code and wish the Postgres devs all the luck in the world to get this working smoothly.
I believe I can safely say that nobody acknowledges and understands the complexity of all state within that class, and that whatever incentives there may be to simplify it are not enough for that to actually happen.
(It ends on line 4692)
I didn't make any claims about the _amount_ of state. My claim was that “you're forced to acknowledge and understand the complexity of your state” (i.e., moving it all together in one place helps understanding the state) is plain-out wrong.
It’s an easy transition path.
Agree.
> It's not like the globals were organized before!
Using a struct with 2000 fields loses some encapsulation.
When a global is defined in a ".c" file (and not exported via a ".h" file), it can only be accessed in that one ".c" file, sort of like a "private" field in a class.
Switching to a single struct would mean that all globals can be accessed by all code.
There's probably a way to define things that allows you to regain some encapsulation, though. For example, some spin on the opaque type pattern: https://stackoverflow.com/a/29121847/163832
A plain global can be accessed from other compiled units - agreed with no .h entry it is my=uch more error prone e.g. you don't know the type but the variables name is exposed to other objects
The `static` modifier in that context just means that the symbol is not exported, so other ".c" files can't access it.
Check out the generated assembly for this simple program, notice that kBase is folded even though it's not marked const: https://godbolt.org/z/h45vYo5x5
But the Postgres mailing list is talking about 2000 global variables being a hurdle to multi-threading. I doubt they just didn't realize that most of them can be optimized into constants.
Merge all that into one process with many threads and it becomes a nightmare problem to ensure some random addon didn't decide to change a global var mid processing (which causes wrong data to be read).
Of course, there's a huge amount of shared space already, so a willful corruption can already do virtually anything. But, more is more.
While we didn't have 2000 globals, we did have a non-trivial amount, spread over about 300kLOC of C++.
We started by just stuffing them into a "context" struct, and every function that accessed a global thus needed to take a context instance as a new parameter. This was tedious but easy.
However the upside was that this highlighted poor architecture. Over time we refactored those bits and the main context struct shrunk significantly.
The result was better and more modular code, and overall well worth the effort in our case, in my opinion.
If anything, it's probably easier to understand.
On the contrary, it's been discussed for ages. But it's a huge change, with only modest advantages.
I'm skeptical of the ROI to be honest. Not that is doesn't have value, but that it has more value than the effort.
+significant and unknown set of new problems, including new bugs.
This reminds me of the time they lifted entire streets in Chicago by 14 feet to address new urban requirements. Chicago, we can safely assume, did not have the option of just starting a brand new city a few miles away.
The interesting question here is should a system design that works quite well upto a certain scale be abandoned in order to extend its market reach.
Years ago I wrote an algorithmic trader in Python (and Cython for the hotspots) using Multiprocessing and I was able to get away with a lot using that approach. I had one process receiving websocket updates from the exchange, another process writing them to an order book that used a custom data structure, and multiple other processes reading from that data structure. Ran well enough that trade decisions could be made in a few thousand nanoseconds on an average EC2 instance. Not sure what their latency requirements are, though I imagine they may need to be faster.
Obviously mutexes are the bottleneck for them at this point, and while my idea might be a bit slower than a low-load situation, perhaps it would be faster when you start getting to higher load.
is this another way to say "in a 2000 member structure, only 10 have significant voting power"?
I'm not. You can get a long way with conventional IPC, and OS processes provide a lot of value. For most PostgreSQL instances the TLB flush penalty is at least 3rd or 4th on the list of performance concerns, far below prevailing storage and network bottlenecks.
I share the concerns cited in this LWN story. Reworking this massive code base around multithreading carries a large amount of risk. PostgreSQL developers will have to level up substantially to pull it off.
A PostgreSQL endorsed "second-system" with the (likely impossible, but close enough that it wouldn't matter) goal of 100% client compatibility could be a better approach. Adopting a memory safe language would make this both tractable and attractive (to both developers and users.) The home truth is that any "new process model" effort would actually play out exactly this way, so why not be deliberate about it?
Or more if a rewrite of subsystems? Like the query planner or storage engine etc?
With regard to client compatibility there are related precedents for this already; the PostgreSQL wire protocol has emerged as a de facto standard. Cockroachdb and ClickHouse are two examples that come to mind.
IPC, to me, includes the conventional shared memory resources (memory segments, locks, semaphores, condition variable, etc.) used by these systems: resources acquired by processes for the purpose of communication with other processes.
I get it though. The most general concept of shared memory is not coupled to an OS "process." You made me question whether my concept of term IPC was valid, however. So what does one do when a question appears? Stop thinking immediately and consult a language model!
Q: Is shared memory considered a form of interprocess communication?
GPT-4: Yes, shared memory is indeed considered a form of interprocess communication (IPC). It's one of the several mechanisms provided by an operating system to allow processes to share and exchange data.
...
Why does citing ChatGPT make me feel so ugly inside?
In that view, I always saw shared memory as IPC, in that it is a tool commonly used to exchange data between processes, but of course it is not strictly tied to any process in particular. This is similar to files, which if you squint are a form of IPC too, and are also not tied to any specific process.
> Why does citing ChatGPT make me feel so ugly inside?
That's probably because, in cases like this, it's not much different to stating it yourself, but is more noisy.
Its the modern let me Google that for you. Just like people don't care what the #1 result on Google is, they also don't care what ChatGPT has to say about it. If they did, they'd ask it themselves.
Postmaster would just share the already shared memory between processes (containing also the locks). That explicit part of memory would opt-in to thread -like sharing and thus get faster/less tlb switching and lower memory usage. While all the rest of the state would still be per-process and safe.
tl;dr super share the existing shared memory area with kernel patch
All operating systems not supporting it would keep working as is.
But I'm very much clueless about internals, so this is a question rather than an opinion.
If you push a lot of work into the database including JSON and have a lot of buffer memory...CPU can easily be limiting.
That said, many of these things are solvable without a full switch to a threaded model (eg. by having pooling built-in and session-state-aware).
Yeeeeesssss, but solving that is solving the hardest part of switching to a threaded model. It requires the team to come terms with the global state and encapsulating session state in a non-global struct.
The thing is that that's a lot easier with threads. Much of the session state lives in process private memory (prepared statements etc), and it can't be statically sized ahead of time. If you move all that state into dynamically allocated shared memory, you've basically paid all the price for threading already, except you can't use any tooling for threads.
I was surprised when starting with postgres. Then again, I have smaller databases (A few TB) and the cache hit ratio tends to be about 95%. Combine that with SSDs, and it becomes understandable.
Even so, I am wary of this change. Postgres is very reliable, and I have no problem throwing some extra hardware to it in return. But these people have proven they know what they are doing, so I'll go with their opinion.
Throw a few NVMe drives at it and it might.
It's orders of magnitude faster than SAS/SATA SSDs and you can throw 10 of them into 1U server. It's nowhere near "slow" and still easy enough to be CPU bottlenecked before you get IO bottlenecked.
But yes, pair of 1TB RAM servers gotta cost you less than half year's worth of developer salary
In those scenarios,there's very little read I/O. CPU is the primary bottleneck. That's why we run up as many as 10 Aurora readers (autoscaled with traffic).
I abuse them for ridiculous things.
Oracle also uses a process model on Linux. At some point (I think starting with 12.x), it can now be configured on Linux to use a threaded model, but the default is still a process-per-connection model.
Why does everybody think it's a bad thing in Postgres, but nobody thinks it's a bad thing in Oracle.
Mainline has a lock so that all backends can use global variables, but only one instance can do codegen at a time.
It was a giant refactoring Especially fun was when multiple compilation units used the same static variable name, but it all worked in the end.
This is the multi-threaded compiler: https://github.com/rsaxvc/tcc-swarm
With the multi-threaded tcc above it scales about as well as multiprocess. With mainline it doesn't scale well at all.
So far I haven't gotten around to reusing anything across libtcc handles/instances, but would eventually like to share mmap()'d headers across instances, as well as cache include paths, and take invocation arguments through stdin one compilation unit per line.
Regarding the threading issue, I think you can do the connections part multithreaded instead of one process per connection and still use IPC between this and postmaster. Because of the way PostgreSQL currently works, seems feasible to move parts one by one into a threaded model and instead of tens/hundreds of processes you can have just a few and a lot of threads.
Honestly, they should prototype it and see how it looks like and then decide on the way forward.
Rewriting in Rust would be interesting, but it would also probably be too invasive to make it worthwile at all - all code in PostgreSQL is C, while not all code in PostgreSQL interacts with the intrinsics of processes vs threads. Any rewrite to Rust would likely take several times more effort than a port to threads.
Then, after more thought, I'm not entirely sure that would be a bad approach. I say this not to advocate for actually rewriting it in Rust, but as a way of describing how difficult this is. I'm not actually sure rewriting the relevant bits of the system in Rust wouldn't be easier in the end, and obviously, that's really, really hard.
This is really hard transition.
I don't think multithread code quality should be measured in absolutes. There are things that are so difficult as to be effectively impossible, which is the lock-based approach that was dominant in the 90s, and convinced developers that it's just impossible difficult, but it's not multithreaded code that's impossibly difficult, it's lock-based multithreading. Other approaches range from doable to even not that hard once you learn the relevant techniques (Haskell's full immutability & Rust's borrow checker are both very solid), but of course even "not that hard" becomes a lot of bugs when scaled up to something like Postgres. But it's not like the current model is immune to that either.
Aside from the trivial while(!transactionSucceeded){retry()} loop, I have trouble proving the correctness of my programs when the number of threads is not small and finite.
— Joe Armstrong
Threads are evil. https://www.sqlite.org/faq.html#q6 https://www2.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-...
Nginx uses an asynchronous event-driven approach, rather than threads, to handle requests. https://aosabook.org/en/v2/nginx.html http://www.kegel.com/c10k.html
They might even reduce complexity that way.
If every process is single threaded, you don't have to consider the possibility of race conditions when accessing any of those ~2000 global variables. And you can pretty much guarantee that little if any of the existing code was written with that possibility in mind.
These are "famous last words" in many contexts, but when talking about difficult-to-reproduce parallelism issues, I just don't think it's a particularly applicable viewpoint at all. No disrespect. :)
Postgres starts from a share-nothing architecture, it’s quite a bit easier to evaluate the addition of sharing.
I don't think that's a fair characterization of the trade offs. Acquiring uncontended mutexes is basically free (and fairly side-effect free) so single-threaded performance will not be noticeably impacted.
Every large C project I'm aware of (read: kernels) that has publicly switched from coarse locks to fine-grained locks has considered it to be a huge win with little to no impact on single-threaded performance. You can even gain performance if you chop up objects or allocations into finer-grained blobs to fit your finer-grained locking strategy because it can play nicer with cache friendliness (accessing one bit of code doesn't kick the other bits of code out of the cache).
1) I don't buy this a priori. Almost everybody who removed a gigantic lock suddenly realizes that there was more contention than they thought and that atomizing it made performance improve.
2) Had Python bitten the bullet and removed the GIL back at Python 3.0, the performance would likely already be back to normal or better. You can't optimize hypothetically. Optimization on something like Python is an accumulation of lots of small wins.
You don’t have to buy anything, that’s been the result of every attempt so far and a big reason for their rejection. The latest effort only gained some traction because the backers also did optimisation work which compensated (and then was merged separately).
> Almost everybody who removed a gigantic lock
See that’s the issue with your response, you’re not actually reading the comment you’re replying to.
And the “almost” is a big tell.
> suddenly realizes that there was more contention than they thought and that atomizing it made performance improve.
There is no contention on the gil in single threaded workloads.
> Had Python bitten the bullet and removed the GIL back at Python 3.0
It would have taken several more years and been completely DOA.
They mentioned a similar issue for Postgres extensions, no?
> Haas, though, is not convinced that it would ever be possible to remove support for the process-based mode. Threads might not perform better for all use cases, or some important extensions may never gain support for running in threads.
Biggest problem of the process model might be the cost of having too many DB connections. Each client need a dedicated server process. Memory usage and the context switching overhead. Or if there is no connection pool, connection time overhead is very high.
This problem has been well addressed with a connection pool. Or having a middle ware instead of exposing the DB directly. That works very well so far.
Oracle has been supporting the thread based model and it's been usable for decades. I remember I tried the thread based configuration option (MTS or shared server) in 1990s. But no one likes that at least within my Oracle DBA network.
It would be a great research project but it would be a big problem if the community pushs this too early.
I've had a similar situation with PHP, where we had written quite a large engine (https://github.com/Qbix/Platform) with many features (https://qbix.com/features.pdf) . It took advantage of the fact that PHP isolated each script and gave it its own global variables, etc. In fact, much of the request handling did stuff like this:
instead of stuff like this: Over the last few years, I have run across many compelling things: It seemed so cool! PHP could behave like Node! It would have an event loop and everything. Fibers were basically PHP's version of Swoole's coroutines, etc. etc.Then I realized... we would have to go through the entire code and redo how it all works. We'd also no longer benefit from PHP's process isolation. If one process crapped out or had a memory leak, it could take down everything else.
There's a reason PHP still runs 80% of all web servers in the world (https://kinsta.com/blog/is-php-dead/) ... and one of the biggest is that commodity servers can host terrible PHP code and it's mostly isolated in little processes that finish "quickly" before they can wreak havoc on other processes or on long-running stuff.
So now back to postgres. It's been praised for its rock-solid reliability and security. It's got so many features and the MVCC is very flexible. It seems to use a lot of global variables. They can spend their time on many other things, like making it byzantine-fault-tolerant, or something.
The clincher for me was when I learned that php-fpm (which spins up processes which sleep when waiting for I/O) is only 50% slower than all those fancy things above. Sure, PHP with Swoole can outperform even Node.js, and can handle twice as many requests. But we'd rather focus on soo many other things we need to do :)
https://symfony.com/doc/current/components/process.html
Going from a working process implementation to async/threads with shared memory is pretty much always a mistake IMHO, especially if it's only done for performance reasons. Any speed gains will be eclipsed by endless whack-a-mole bug fixes, until the code devolves into something unrecognizable. Especially when there are other approaches similar to map-reduce and scatter-gather arrays where data is processed in a distributed fashion and then joined into a final representation through mechanisms like copy-on-write, which are supported by very few languages outside of PHP and the functional programming world.
The real problem here is the process spawning and context-switching overhead of all versions of Windows. I'd vote to scrap their process code in its entirety and write a new version based on atomic operations/lists/queues/buffers/rings with no locks and present an interface which emulates the previous poor behavior, then run it through something like a SAT solver to ensure that any errata that existing software depends on is still present. Then apps could opt to use the direct unix-style interface and skip the cruft, or refactor their code to use the new interface.
Apple did something similar to this when OS X was released, built on a mostly POSIX Darwin, NextSTEP, Mach and BSD Unix. I have no idea how many times Microsoft has rewritten their process model or if they've succeeded in getting performance on par with their competitors (unlikely).
Edit: I realized that the PHP philosophy may not make a lot of sense to people today. In the 90s, OS code was universally terrible, so for example the graphics libraries of Mac and Windows ran roughly 100 times slower than they should for various reasons, and developers wrote blitters to make it possible for games to run in real time. That was how I was introduced to programming. PHP encapsulated the lackluster OS calls in a cross-platform way, using existing keywords from popular languages to reduce the learning curve to maybe a day (unlike Perl/Ruby, which are weird in a way that can be fun but impractical to grok later). So it's best to think of PHP more like something like Unity, where the nonsense is abstracted and developers can get down to business. Even though it looks like Javascript with dollar signs on the variables. It's also more like the shell, where it tries to be as close as possible to bare-metal performance, even while restricted to the 100x interpreter slowdown of languages like Python. I find that PHP easily saturates the processor when doing things in a data-driven way by piping bytes around.
TLBs are "just a detail" of roughly 100% of server, desktop, and mobile CPUs.
> Also, aren't they per core and not per process? What would that solve then to switch to MT?
TLB entries are per address space. Threads share an address space, processes do not.
IMO: if the variable is _truly_ global, i.e. code all over the codebase cares about it, then it should just be global instead of pretending like it's not with some fancy architecture.
The tricky part is reacting to changes to a global variable. Writing a bunch of "on update" logic leads to madness. The ideal solution is for there to be some sort of one-directional flow for updates, like when a React component tree is re-rendered... but that's very hard to build in an application that doesn't start out using a library like React in the first place.
Robert Haas pointed out in one message that an implementation pattern was to use things like file-scope static storage duration variables to provide session-local state for individual components. This is why they've been arguing against a single giant structure declared in "session.h" as an approach, as it requires every future addition to session state to touch the central core of the entire program.
They want to keep the advantage of the fact that these variables are in fact not global. They are local; and the problem is rather that they have static storage duration and are not per-thread, and thus are not per-session in a thread-per-session model.
If someone didn't take the time to update the comments and the reviewers didn't point it out, then you've probably found the bug because someone was cowboying some shitty code.
Outdated comments are often way worse than no comments, because they can give you wrong ideas that aren't true anymore, and send you off in the wrong direction before you finally figure out the comment was wrong.
enum kinds { writers; readers; updaters; }
On UNIX systems, Oracle uses a multi-process model, and you can see these:
Windows forks processes about 100x slower than Linux, so Oracle runs threaded on that platform in one great big PID.Sybase was the first major database that fully adopted threads from an architectural perspective, and Microsoft SQL Server has certainly retained and improved on that model.
I work with a Windows-based COTS webapp that uses Postgres w/o any connection pooling. It's nearly excruciating to use because it spins-up new Postgres processes for each page load. If not for the fact that the Postgres install is "turnkey" with the app I'd just move Postgres over to a Linux machine.
We're really talking about X-per-client as the primary reason to move away from processes, right?
So if you can get most of the benefit via pooling... why inherit the pain of porting?
Presumably latency jitter would be a difficult problem with pools, but it seems easier (and safer) than porting processes -> threads.
Disclaimer: High performance / low latency DB code is pretty far outside my wheelhouse.
Many other things too. Like better sharing of caches. Lower overhead of thread instead of process. Etc. (read the thread)
Could you point out, aside from the large numbers of clients I mentioned (and the development overhead of implementing multi-process memory management code), what the article mentions is a primary drawback of using processes over threads?
Probably still requires the parameter to be set.
Oracle - never had a scalability issue on very big Linux, Solaris and HPUX systems though - they do it well in my experience.
It's optional, and the default is still a process model on Linux.
The other thing that might be interesting is FUTEX_SWAP / UMCG. Although it doesn't remove the overhead induced by context switches entirely (specifically, you would still deal with TLB misses), you can avoid dealing with things like speculative execution exploit mitigations.
The challenge with this setup is that you need to do 1 writer and multiple reader clusters so you end up doing localhost replication which is super weird. If that requirement was somehow removed that’d be awesome for scaling really huge clusters.
On the other hand, having the ability to shut down and cleanup the entire memory space of a single connection by just disconnecting is really nice, especially if you have extensions that do interesting things.
> Tom Lane said: "I think this will be a disaster. There is far too much code that will get broken". He added later that the cost of this change would be "enormous", it would create "more than one security-grade bug", and that the benefits would not justify the cost.
That the all of those would work faster because of performance benefits, as mentioned in article
Anyway, I think there are definitely limits that are starting to appear with Postgres in some spots. This is probably one of the most difficult possible solutions to some of those problems, but even if they don't switch to a fully threaded model, being more CPU efficient, better connection handling, etc will all go a substantial way. Doing some of the really hard work is better than none of it, probably.
If Tom Lane says it will be a disaster, I believe it will be a disaster.
But there is another way.
Ok?
[0] https://www.thegeekdiary.com/what-is-sql-server-operating-sy...
I think "Idiom: a tic (Rust)" can also fit if I squint hard enough and decide it looks like a definition from an online dictionary :-)
Ignoring the above, first state the explicit requirements driving this change and let people weigh in on those. This sounds like a geeky dev itch.
For those who don't follow PHP closely - that version was an attempted refactor of the string implementation which essentially shut down nearly all work on PHP for a decade, stagnating the language until it became pretty terrible compared to other options. They finally gave up and started work on PHP 7 which uses the (perfectly good) PHP 5 strings.
Ten years of wasted time by the best internal PHP developers crippled the project - I'm amazed it survived at all.
I got my real professional start using PHP, and have built even financial systems in the language (since ported to .NET 6 for my ease of maintenance, and better number handling). I'm still very interested in the language itself, in case I ever have the need to freelance or provide a solution to a client that can't afford what I can build in .NET (although to be honest, at this point I'm roughly able to code at the same speed in .NET as in PHP, but with the added type-safety, although I know PHP has really stepped up in providing this).
Regular PHP strings are actually pretty great, since you can treat them like byte arrays. Fun fact: PHPs streaming API has an “in-memory” option and it’s… just a string under the hood.
Just don’t forget to use multibyte functions when you’re handling things like user input.
Weird book IMO, because it has a lot of content that's just about general software development, rather than anything to do with PHP specifically, or the theoretical PHP6 APIs in particular.
https://www.joelonsoftware.com/2000/04/06/things-you-should-...
An oldie but a goodie
You should rewrite code only when the cost of adding a new feature (one that is actually necessary) to the old codebase becomes comparable to designing your entire system from scratch to allow for that feature to be added easily. That is to say that the cost of the rewrite should become comparable to the cost of continuing development. I have been a part of a couple of rewrites like that, one of them quite complex, and yes they were warranted and yes they worked.
But having said that you should absolutely be conservative with rewriting code. It’s a bad habit to always jump to a rewrite.
I don’t think you necessarily need to be conservative about rewriting things. We do it all the time in fact. We build something to get it out there and see the usage, and then we build it better and then we do it again. Which often involves a lot of “rewriting” but thanks to principles like SOLID’s single responsibility makes this rather easy to both do and maintain (we write a lot of semi-functional code and try to avoid using OOP unless necessary, so we don’t really use all the parts of SOLID religiously).
I do agree that it’s never a good idea to get into things with the mind-set of “we can do this better if we start from scratch” because you can’t.
Why not identify that same logical feature and move it into a library. How does a Microservice add value here?
Identifying, extracting and adding tests to logical features has been the sane way to rewrite software for ages. Michael feathers even wrote a book about it [1]. This ship of theseus approach works because it's incremental and allows for a complete rewrite without ever having non-functional software in between.
Adding a REST/networking/orchestration boundary to make it a Microservice just for the sake of extracting it adds a lot of complexity for no gain.
Microservice can be the right architecture, but not if all you want is to extract a library.
[1] https://understandlegacycode.com/blog/key-points-of-working-...
You shouldn't stop development on your important products.
Letting a couple of your talented programmers loose on a greenfield reimplementation is a perfectly sane strategic move.
Stopping development on important products because you are 100% certain that the reimplementation will be successful by $DEADLINE is a foolish gamble.
Between the 3 P's, Python's strategic decisions in 2000s were clearly the most successful.
And it wasn't a total rewrite.
So in order for there to be a net gain, the gang working on the alternative have to be able to find such big wins as to being neigh impossible.
This is a very very hard problem in our domain. 99% of the time, we have to simply resist the urge to _just rewrite the sucker_. No! Don't do it! (And this is incredibly hard because we all want to.)
Say you have product A with features P, Q, R, and S; writing product B has to reproduce P, Q, R, and S, plus X and Y that is currently produced by the A team. On top of that, it has to fix (conceptual) bugs within P, Q, R, and S. All this is to be done by the new, crack, team that aims to make it so that: cost-of-development(B) < cost-of-development(A).
But the point is that the difference in magnitude of cost-of-development(B) and cost-of-development(A) has to be rather large considering the amount of work needed to have a return on that investment at all.
> They decided to rewrite the code from scratch.
Absolutely not what's being proposed for Postgres.
In practical terms, if every line needs to be audited and updated, it is a re-write
Experience with other systems has taught me that in a system that's been in active use and development for decades, entanglement will be deep, subtle, and pervasive. If this isn't true of postgres then it's an absolute freak anomaly of a codebase. It is that in other ways, so it's possible.
But the article mentions there being thousands of global variables. And Tom Lane himself says he considers it untenable for exactly this reason. That's a very good reason to think that it will require that many changes imo.
The 'that many changes' in question is a complete rewrite. Many changes across many files, yes, but nothing even approaching a rewrite.
> I don't have any particular knowledge of postgres
Judging by their bio, the person you're replying to does.
A large refactor at best. It will touch lots of parts of the code base, but the vast majority of the source code would remain intact. Otherwise they could just Rewrite it in Rust™ while they’re at it
> if every line needs to be audited and updated, it is a re-write
I’m not sure why you believe every line needs to be updated. Most code is thread agnostic.
One of the main programmers suggested making it so that the next guidance routine wouldn't run until the previous one was done. This would make the code less sensitive to race conditions and allow more useful functionality for the pilots (who were the actual users and did seem to want it). However everyone assumed the two-second constant was implicitly embedded everywhere.
It wasn't -- only in a few places -- and with that fixed the code got more general and the proof of concept ran better than ever in about every simulator available. The amount of control it gave pilots was years ahead of the curve. But it never got a chance to fly on a real mission because what was there was "good enough" and nobody bothered to try.
In our combined comments there's a lesson about growing experiments and figuring out how to achieve failure quickly.
I'm not trying to naysay it per se, bc again I don't have technical knowledge of this codebase. But that's exactly the sort of scenario that can cause a large project to splinter or stall for years. Talking about "the implementation" absent the context that would be necessary to create that implementation seems naively optimistic, or at worst irresponsible.
They could fork, they could add threading to some sub systems and roll it out over several versions.
I don't know enough about the code but, of course, it is a hard problem but the solution might be to build it from the ground up as a threaded system, using the skills learned over 30 years and taking the hit on the rebuild instead of reworking what is there.
I am most interested because I didn't realise there was a performance problem in the first place.
Going the other way, there is an extremely well understood interface between all the processes which run in isolation: shared memory. Nearly by definition this must be well coordinated between the processes.
So the first step in moving to a multi-threaded implementation would be to change nearly nothing about each process, and then just run each process in its own pthread, keeping all the shared memory ‘n all.
You would expect performance to be about the same, maybe a little better with the reduces TLB churn, but the architecture is basically unchanged. At that point, you can start to look at what are more appropriate communication/synchronisation mechanisms now you’re working in the same address space.
I just don’t understand why so many people seem to think this requires an enormous rewrite - having developed as a multi-process system means you’ve had to make so much of the problematic things explicit and control for them, and none of these threads would know anything at all about each other’s internals.
Without being familiar with the Postgres source, this seems to be what I call a "somersault problem": hard to break down into sub-goals. I have heard that the Postgres codebase is solid which makes it easier but it's still mature and highly complex. It doesn't sound feasible to me.
https://kristiandupont.medium.com/somersault-problems-69c478...
Why not work on something like that instead of changing something that works? Especially since they the process model really only runs into trouble on large systems.
If you want to have a software _engineering_ discussion about the trade offs involved in sharing global mutable state, this is a good venue for that.
All engineering is trade offs. As soon as you start telling people they’re making blanket statements that X is always bad, you’ve transitioned into the realm of nitpicking.
PostgreSQL uses a process model. So the article just states that globals work fine for PostgreSQL
> Knizhnik has already done a threads port of PostgreSQL. The global-variable problem, he said, was not that difficult.
I see no big problem based on information from person who did some porting already.
The fact is sometimes it's good. Saying it's universally bad is going beyond the realm of logic and evidence and into the realm of ideology.
If you want to contribute to the discussion, I'd be happy to be given an example of successful usage of global variables that made a project a long term success under changing requirements compared to the alternatives.