Re: Further pg_upgrade analysis for many tables

On Sun, Jan 20, 2013 at 02:11:48PM -0500, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > ! * Using pg_restore --single-transaction is faster than other
> > ! * methods, like --jobs.
>
> Is this still the case now that Jeff's AtEOXact patch is in? The risk
> of locktable overflow with --single-transaction makes me think that
> pg_upgrade should avoid it unless there is a *really* strong performance
> case for it, and I fear your old measurements are now invalidated.

I had thought that the AtEOXact patch only helped single transactions
with many tables, but I now remember it mostly helps backends that have
accessed many tables.

With max_locks_per_transaction set high, I tested with the attached
patch that removes --single-transaction from pg_restore. I saw a 4%
improvement by removing that option, and 15% at 64k. (Test script
attached.) I have applied the patch. This is good news not just for
pg_upgrade but for other backends that access many tables.

git patch
1 11.06 11.03
1000 19.97 20.86
2000 28.50 27.61
4000 46.90 45.65
8000 79.38 80.68
16000 153.33 147.13
32000 317.40 302.96
64000 782.94 659.52

FYI, this is better than the tests I did on the original patch that
showed --single-transaction was still a win then:

http://www.postgresql.org/message-id/20121128202232.GA31741@momjian.us

> #tbls git -1 AtOEXAct both
> 1 11.06 13.06 10.99 13.20
> 1000 21.71 22.92 22.20 22.51
> 2000 32.86 31.09 32.51 31.62
> 4000 55.22 49.96 52.50 49.99
> 8000 105.34 82.10 95.32 82.94
> 16000 223.67 164.27 187.40 159.53
> 32000 543.93 324.63 366.44 317.93
> 64000 1697.14 791.82 767.32 752.57

Keep in mind this doesn't totally avoid the requirement to increase
max_locks_per_transaction. There are cases at >6k where pg_dump runs
out of locks, but I don't see how we can improve that. Hopefully users
have already seen pg_dump fail and have adjusted
max_locks_per_transaction.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +