Virtually any time you put a fixed sleep in your program, it's going to be wrong. It is either too long or too short, and even if it is perfect, there is no guarantee your program will actually sleep for exactly the requested amount of time. A condition variable or something similar would be the right thing to use here.
Of course, if this code path really is only taken very infrequently, you can get away with it. But assumptions are not always right, it's better to make the code robust in all cases.
The tl;dr is that Postgres, as any long-running "server" process (especially as a DBMS server!) does not run with SIG_DFL as the handler for SIGTERM; it instead sets up the signal handler that merely records the fact that the signal has happened, in hopes that whatever loops are going on will eventually pick it up. As usual, some loops don't but it's very hard to notice.
Fascinating root cause: a missing CHECK_FOR_INTERRUPTS() left pg_create_logical_replication_slot basically unkillable on hot standbys. Simple fix, but huge impact.
Makes me wonder how many other Postgres processes might ignore SIGTERM under edge conditions. Do folks here test signal handling during failovers or replica maintenance? Seems like something worth adding to chaos tests.
> While the Postgres community has an older and sometimes daunting contribution process of submitting patches to a mailing list, [...]
The Linux kernel works the same way.
What are other big open source projects using if they are not using mailing lists?
I could imagine some using GitHub but IME it's way less efficient than mailing list. (If I were a regular contributor to a big project using GitHub, me being me, I would probably look for or write myself a GitHub to e-mail gateway).
You can already subscribe to projects or single issues/PRs on github and reply via email to post comments.
I can understand not wanting to use GitHub/GitLab/etc. for various reasons. But I don't understand how usability vs mailing lists is one.
How is a set of 9+ mailing lists any better? It has significantly worse discovery and search tools unless you download all the archives. So you're creating a hurdle for people there already.
Then you have people use all kinds of custom formatting in their email clients, so consistent readability is out the window.
People will keep top-posting (TOFU), transforming the inconsistent styles in the process. Creating an unnecessarily complicated problem for your email client to "detect quotes", or you have to keep reminding people.
Enforcing structure of any kind in email lists seems so tedious. I'm not advocating for bugzilla style "file out these 20 nonsensical fields before you can report anything" but some minimal structure enforced by some tooling as opposed to manual moderation seems very helpful to me.
Good find! I've seen similar behavior before and was wondering why it wasn't easy to stop.
This isn't the only place Postgres can act like this, though. I've seen similar behavior when a foreign data wrapper times out or loses connection, and had to resort to either using kill -9 or attaching to the process using a debugger and closing the socket, which oddly enough also worked.
Might be worth generalizing this approach to also handle that kind of failure
I’ve seen this behavior (backend processes don’t honor pg_terminate_backend(pid); ), albeit in an older version (PostgreSQL 14), and it was caused by excessively long plans for the LLVM-based JIT. Disabling the JIT fixed it.
11 comments
[ 3.9 ms ] story [ 19.1 ms ] threadOf course, if this code path really is only taken very infrequently, you can get away with it. But assumptions are not always right, it's better to make the code robust in all cases.
Makes me wonder how many other Postgres processes might ignore SIGTERM under edge conditions. Do folks here test signal handling during failovers or replica maintenance? Seems like something worth adding to chaos tests.
The Linux kernel works the same way.
What are other big open source projects using if they are not using mailing lists?
I could imagine some using GitHub but IME it's way less efficient than mailing list. (If I were a regular contributor to a big project using GitHub, me being me, I would probably look for or write myself a GitHub to e-mail gateway).
I can understand not wanting to use GitHub/GitLab/etc. for various reasons. But I don't understand how usability vs mailing lists is one.
How is a set of 9+ mailing lists any better? It has significantly worse discovery and search tools unless you download all the archives. So you're creating a hurdle for people there already.
Then you have people use all kinds of custom formatting in their email clients, so consistent readability is out the window.
People will keep top-posting (TOFU), transforming the inconsistent styles in the process. Creating an unnecessarily complicated problem for your email client to "detect quotes", or you have to keep reminding people.
Enforcing structure of any kind in email lists seems so tedious. I'm not advocating for bugzilla style "file out these 20 nonsensical fields before you can report anything" but some minimal structure enforced by some tooling as opposed to manual moderation seems very helpful to me.
This isn't the only place Postgres can act like this, though. I've seen similar behavior when a foreign data wrapper times out or loses connection, and had to resort to either using kill -9 or attaching to the process using a debugger and closing the socket, which oddly enough also worked.
Might be worth generalizing this approach to also handle that kind of failure