Hot Standby 0.2.1

OK, here is the latest version of the Hot Standby patchset. This is
about version 30+ by now, but we should regard this as 0.2.1
Patch against CVS HEAD (now): clean apply, compile, no known bugs.

OVERVIEW

You can download PDF versions of the fine manual is here
http://wiki.postgresql.org/images/0/01/Hot_Standby_main.pdf
http://wiki.postgresql.org/images/0/01/Hot_Standby_Recovery_Functions.pdf

Also available via the project Wiki, which is here
http://wiki.postgresql.org/wiki/Hot_Standby

Patch should be attached to this email. If problems, get this and other
versions from Wiki please. No offlist comments, questions etc please.

PATCH VERSIONING & STATUS

0 meaning its not fully released yet
2 meaning this is a major new re-write
1 meaning this is the first release

Patch is still in testing and will be for next few days at least.
Released now only because I promised to do so.

Is this ready for commit? Yes, it is in the shape I want it to be in,
but also, No, I can't say it's been through a wide enough range of tests
as yet to be considered immediately ready for commit.

Further bug fixing and minor cosmetic development will take place via my
GIT repo, uploaded soon. Patch included here to meet deadlines and code
inclusion. BSD.

PATCH SUMMARY

* 76 files changed, 5160 insertions(+), 59 deletions(-), 1251 mods(!)
* 7 files with more than 100 lines changed
procarray.c (1200+ additions)
xlog.c (600+ additions)
xact.c (500+ additions)
inval.c (650 additions)
lock.c
heapam.c
nbtxlog.c
* 29 files with 10 or fewer lines changed
* Applies cleanly to CVS HEAD as of now

CHANGES

The rough changes since version 1 series of patches.

* Full documentation included. Many, but not all nuances of SGML tagging
followed, but sufficient aspects there to allow for proofreading before
we do final changes. Some undocumented functions now correctly
documented. Recovery functions now split into user and control functions
in docs to make it clearer.

* Starting conditions in GetRunningTransactionData() are now much
stricter and holds more lwlocks. There are few cases where any not-found
xids are allowed during xid processing, so code is more robust. Please
check for race conditions.

* GetRunningTransactionData() now handles initialisation of
AccessExclusiveLocks correctly. Locks are counted in a low-contention
approach that avoids taking holding lock partition locks, if possible.

* Start-up conditions now recoded to allow faster start in cases where
many subtransactions are present. Recovery connections are only enabled
when the snapshot is valid.

* max_connections needs to be correctly set or HS will not allow
connections. Once snapshots are enabled they will continue to be
available always.

* RecordKnownAssignedTransactions() now contains a test for xid
wraparound threat which invokes conflict processing should that occur.

* Boolean states now clarified and corrected. Hot Standby can be turned
off completely if not required or if problems effect production. That
causes many changes but there is no change in the intention of those
sections of code.

* UnobservedXids processing follows Heikki's proposal, but has been
renamed to KnownAssignedXids. It has also been modularised and
completely re-written using a hash table approach. So far it has been
much more stable than the previous sorted array coding, which I am happy
to see in the shredder for all the problems it caused. Fully detailed
comments all through.

* All record types now respect max_standby_delay.

* All deferred conflict processing has been removed - conflict
processing itself is still enabled.

* A few other functions have been renamed and/or moved around to
rationalise their exact purpose/position within their modules.

* Prepared transactions holding AccessExclusiveLocks at the end of
recovery are now handled.

* Hash indexes are now safely handled. That was removed at request, but
we need it to avoid silent data loss for queries near those types of
index.

* Hint bits are now set, in appropriate circumstances.

* Flat file logic removed

* Large swathes of unused code removed.

* All code comments addressed and/or re-explained in more detail

* CHECKPOINT now works during recovery but performs restartpoint
instead.

* Tweaked max_standby_delay code to avoid long duration waits. Added
dynamic function to control delay during recovery. Added code for stats
collection and ps display. Set default to sensible production values.

RECENT BUGS

* Found and fixed missing relcache init file invalidation

* Found and fixed more serious VACUUM FULL-related weirdness <sigh>

* Recently discovered bug has resulted in changes in
XidInMVCCSnapshot(). Heikki's earlier approach did not correctly allow
for the maximum size of a snapshot. The simplicity of Heikki's proposal
is good, but hid a flaw in where snapshots would put their xids. I've
fully solved the problem though I expect further discussion.

I've looked through every change and verified it, but fixing all the
bugs means there's areas of new code added in last few days. I accept
that any bugs herein are my responsibility.

--
Simon Riggs www.2ndQuadrant.com