Parallel query will probably win for sexiest new feature (rightfully so!) but remote apply is very useful in the real world. It will greatly simplify reading newly written data off replicas: http://michael.otacoo.com/postgresql-2/postgres-9-6-feature-...
Postgres currently uses a single process per connection. That means at most one CPU core is used per query executing a connection. Parallel query allows breaking up query plans to allow multiple processes (and by extension cpu cores) to process a part of a query plan.
The classic example is a sequential scan, say for a COUNT(*) or SUM(foo). A single process would scan through all the blocks in order. A parallel scan would split the blocks into N buckets, scan them via N separate processes, and then combine the results.
There's nothing to complain about with this. But aren't most large database queries ultimately IO capped, rather than CPU capped? Wouldn't this just put more load on the IO and it would just bottleneck there?
Yes and no. Take the following with grain of salt as it is just my personal opinion/understanding. Only 1 or 2 points are actually from challenges I have run into.
Restricting every query to a single core places an artificial CPU cap on a system that would otherwise be IO capped.
As analytics keeps gaining popularity, the need/desire to move more and more CPU bound processing to the data storage layer is increasing.
And finally, PostgreSQL's advanced query planning/partitioning support would immensely benefit from being able to split partition queries into multiple process buckets.
All depends on the workload, aggregates are more often CPU capped than IO cap in workloads we use in-house. We have a PostgreSQL server with 16GB of memory (the database total is over 80GB, but the table that is hit the most is under 3GB, the rest is just BLOB's) - more than enough to keep the relevant data in memory. Most of the time running some aggregate queries that hit the database server pretty hard are spent in USER, not IOWAIT. It's pretty annoying, because I frequently see my CPU usage at 50% (one of two cores maxed) when I know these queries could be parallelized and take advantage of the second core (or more).
Now queries can be parallelized, combining the results of several subqueries in different processes. This is a stepping stone to being able to run several subqueries on different machines. And if those different machines contain different shards of data, well, PostgreSQL has just evolved into a scale out 'Big Data' database without throwing out any relational features. Two phase commit, synchronous replication, logical replication - all the pieces are falling into place, and you can see the remaining pieces forming in extensions like BDR and Citrus Data and others.
>The age-old assumption that I/O is slow and computation is fast is no longer true: this invalidates decades of design decisions that are deeply embedded in today’s systems
The Postgres beta periods are usually really, really, really long. Mid/End may is traditionally when the first beta comes out for a release in late autumn.
Especially for software so mission critical as a RDBMS. If your web framework has a bug users get errors or things act wonky, if your database has a bug you can get data corruption and other horrible, horrible things.
UPSERT from previous release plus
Parallel sequential scans, joins and aggregates and
Full text search for phrases once 9.6 is ready
will make PG almost perfect :)
This is excellent, the read-balancing consistency is a huge feature, and one we've been waiting for as trying to do read balancing without it is jumping through way too many hoops (unless data consistency just isn't your thing).
I love new features in Postgres, but I hate upgrading. Transition, especially for dev environment is always way too complex and not obvious at all.
If there is one feature I wish PG had, it is recognizing old format db and offering one command that can in place upgrade damn db, without me googling it every time.
> If there is one feature I wish PG had, it is recognizing old format db and offering one command that can in place upgrade damn db, without me googling it every time.
So your having to Google for five minutes when doing a major version upgrade is the biggest flaw in modern postgres? Oh come on now.
P.S. pg_upgradecluster in Debian would probably do this for you without googling
Hey, I know it sounds a little spoiled, but when I am focused on something else, especially in development context, I could use some more automated way.
I am sorry you got downvoted, but as you can see, it is a genuine pain that software should do for us. On other hand, PG is fantastic DB and it is great I have this as a problem.
pg_upgrade is a pretty reasonable compromise, but it would be nice if an upgrade didn't require your entire database be taken offline for the duration. I can install a new MS SQL Server instance, attach a database from the old instance and be going in under a minute of downtime - this is literally the only pain point I have with Pq, but considering I don't typically upgrade EVERY release (I still have 9.4 instances that have no reason to be upgraded to 9.5 since the applications aren't in-house) it's not typically a huge deal.
pg_upgrade in link mode typically takes less than a minute to run unless you have a lot of tables; the runtime is dependant on the number of database objects, not rows or size of the database.
Database statistics are not preserved across pg_upgrade so you'll have to run ANALYZE on the database after the upgrade which may take a while as it usually needs to read all or almost all of your data.
I just recently upgraded a ~1 TB database, the actual upgrade process took less than a minute but analyzing it took much longer.
currently searching for 'new york' would match 'york new', as the parser doesn't understand phrases, but new version would let you match '"new york"' to 'new york' but not 'york new'
You have to enable it by configuring the maximum number of workers per query, and then the query planner tries to figure out if the query would benefit from a parallel execution. This is the first release with parallelism so I would not be surprised if the query planner gets this wrong in some cases.
The features that I'll personally probably enjoy the most are really the FDW join pushdown work, parallel query. (though it's still early days for the parallel stuff)
..and the thing that's a killer feature for many larger databases, the freeze map. Not having to FREEZE things all the time is a godsend.
To avoid having to vacuum whole relations all the time there's the "visibility map", which keeps track of which blocks have no dirty rows; so they can be skipped during vacuum.
To understand which versions are visible postgres stores transaction-ids in the row headers; these are four byte wide. On systems with significant throughput, those don't last very long. So every now and then there's anti-wraparound vacuums; which replaces xids which are about to become "too old" with a special marker ("frozen").
The problem < 9.6 is that these anti-wraparound vacuums have to scan the whole table, thereby are a lot more expensive. These happen every 200 million xids by default (write transactions and/or savepoints), configurable up to ~2 billion. With the freeze map, that doesn't have to happen, only non-frozen pages have to be re-scanned.
Is there something specifically interesting in the 9.6 release that makes it more noteworthy than other releases? I enjoy reading PostgreSQL release announcements, as I myself haven't used it but have plans to do some testing with is when some time free up, but I'm a bit confused as to why a beta release announcement is getting so much attention.
I guess because beta usually determines which features are going to be in the final release. So if you're looking what release to choose for a new version of your software, it might be more worthwhile to look at beta than the final release.
41 comments
[ 3.1 ms ] story [ 101 ms ] threadMajor enhancements in PostgreSQL 9.6 include:
Parallel sequential scans, joins and aggregates
Elimination of repetitive scanning of old data by autovacuum
Synchronous replication now allows multiple standby servers for increased reliability
Full-text search for phrases
Support for remote joins, sorts, and updates in postgres_fdw
Substantial performance improvements, especially in the area of improving scalability on many-CPU servers
The classic example is a sequential scan, say for a COUNT(*) or SUM(foo). A single process would scan through all the blocks in order. A parallel scan would split the blocks into N buckets, scan them via N separate processes, and then combine the results.
Restricting every query to a single core places an artificial CPU cap on a system that would otherwise be IO capped.
Disks/storage is also getting faster (http://www.geek.com/chips/new-intel-storage-is-1000-times-fa...).
As analytics keeps gaining popularity, the need/desire to move more and more CPU bound processing to the data storage layer is increasing.
And finally, PostgreSQL's advanced query planning/partitioning support would immensely benefit from being able to split partition queries into multiple process buckets.
>The age-old assumption that I/O is slow and computation is fast is no longer true: this invalidates decades of design decisions that are deeply embedded in today’s systems
https://blog.acolyer.org/2016/01/15/non-volatile-storage/
But this is just a start and there are lots more opportunities to improve as mentioned here: https://wiki.postgresql.org/wiki/Parallel_Query_Execution
http://www.postgresql.org/docs/9.6/static/release-9-6.html
* [ "asynchronous and vectorized execution" ]
http://www.postgresql.org/message-id/CA+Tgmobx8su_bYtAa3Dgrq...
If there is one feature I wish PG had, it is recognizing old format db and offering one command that can in place upgrade damn db, without me googling it every time.
So your having to Google for five minutes when doing a major version upgrade is the biggest flaw in modern postgres? Oh come on now.
P.S. pg_upgradecluster in Debian would probably do this for you without googling
That being said, if upgrade is the biggest problem, that's great news!
I am sorry you got downvoted, but as you can see, it is a genuine pain that software should do for us. On other hand, PG is fantastic DB and it is great I have this as a problem.
http://www.postgresql.org/docs/9.5/static/pgupgrade.html
Database statistics are not preserved across pg_upgrade so you'll have to run ANALYZE on the database after the upgrade which may take a while as it usually needs to read all or almost all of your data.
I just recently upgraded a ~1 TB database, the actual upgrade process took less than a minute but analyzing it took much longer.
..and the thing that's a killer feature for many larger databases, the freeze map. Not having to FREEZE things all the time is a godsend.
To avoid having to vacuum whole relations all the time there's the "visibility map", which keeps track of which blocks have no dirty rows; so they can be skipped during vacuum.
To understand which versions are visible postgres stores transaction-ids in the row headers; these are four byte wide. On systems with significant throughput, those don't last very long. So every now and then there's anti-wraparound vacuums; which replaces xids which are about to become "too old" with a special marker ("frozen").
The problem < 9.6 is that these anti-wraparound vacuums have to scan the whole table, thereby are a lot more expensive. These happen every 200 million xids by default (write transactions and/or savepoints), configurable up to ~2 billion. With the freeze map, that doesn't have to happen, only non-frozen pages have to be re-scanned.