Re: WAL format and API changes (9.5)

On Tue, Nov 18, 2014 at 4:31 AM, Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
wrote:
> WAL insertion performance
> =========================
> To measure the performance of generating WAL, I ran the
> wal-update-testsuite.sh that Amit also ran earlier. The cluster was
> initialized with:
>
> shared_buffers=512MB
> checkpoint_segments=30
> fsync=off
> autovacuum=off
> full_page_writes=off
>
> [results]
> Summary: No meaningful difference in runtime.
If I am seeing that correctly, WAL generated is reduced for all the tests,
except for the case of "hundred tiny fields" where more WAL is generated.
Now the duration time seems to be generally reduced, some noise (?) making
it sometimes higher.

> WAL replay performance
> ======================
>
> To test WAL replay performance, I ran pgbench with WAL archiving enabled,
> and timed the replay of the generated WAL. I used the attached script,
> replay-perf-test.sh for that. full_page_writes were disabled, because
> replaying full page images is quite different from replaying other
records.
> (Performance of full-page images is interesting too, but it's not expected
> that these WAL format changes make much difference to that).
>
> In summary, there is no significant difference in replay performance. The
> amount of WAL generated is much smaller with the patch.

> This concludes my performance testing, until someone wants to see some
other
> scenario being tested. I'm happy with the results.
I think you can, that's a great study, and this proves to be a gain on many
fields.

If this goes in, it is going to be one of the largest patches committed
ever.
$ git diff --stat | tail -n1
91 files changed, 3895 insertions(+), 4305 deletions(-)

There are still some XXX blocks here and there in the code.. But nothing
really huge, like here:
- * checks could go wrong too.
+ * checks could go wrong too. XXX does this comment still
make sense?
*/
- Assert(xldata->blkno != xldata->blknoNew);
+ Assert(blkno != blknoNew);

Btw, did you do a run with the buffer capture facility and checked for page
differences?

Except that, after going through the code once again, ISTM that the patch
is in a nice state. It may be better to wait for some input from Andres, he
may catch some issues I haven't spotted.
Regards,
--
Michael

On 11/18/2014 10:28 AM, Michael Paquier wrote:
> Btw, did you do a run with the buffer capture facility and checked for page
> differences?

Yes. That's how I bumped into the bug fixed in c73669c0. That tool has
been tremendously helpful.

> Except that, after going through the code once again, ISTM that the patch
> is in a nice state. It may be better to wait for some input from Andres, he
> may catch some issues I haven't spotted.

Yeah, I'm sure there are tons of small things. I'll have to take a small
break and read through the patch myself after a few days. But I'd like
to get this committed pretty soon. I believe everyone agrees with the
big picture, even if there's some little tweaking left to do. It's a
pain to maintain as a patch, and I believe the FPW compression patch is
waiting for this to land.

- Heikki