Minor changes to Recovery related code

Lists: pgsql-hackerspgsql-patches
From: "Simon Riggs" <simon(at)2ndquadrant(dot)com>
To: <pgsql-hackers(at)postgresql(dot)org>
Subject: Minor changes to Recovery related code
Date: 2007-03-30 10:20:04
Message-ID: 1175250004.4386.723.camel@silverbirch.site
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

I'd like to make the following changes to recovery related code over the
next few days/weeks. If anybody insists I do this by freeze or not at
all, then I'll submit patches for 1,3,4,5,10 before Saturday night. I'd
rather take a bit more time and do this in one drop and there are some
code dependencies between these changes and other patches from
Koichi-san and myself.

1. Current xlog should be archived at shutdown for smart shutdown
- check that archiver is active prior to shutdown checkpoint request
- if (shutdown checkpoint && XLogArchivingActive()) RequestXLogSwitch()
- for smart shutdown, have archiver complete its work before exiting

2. pg_stop_backup() should wait until all archive files are safely
archived before returning

3. Need a %r parameter for restore_command, to allow the restore command
be passed the name of the file containing the last restartpoint. This
will allow the restore command to clean down old archive files more
safely/cleanly in Warm Standby operation.
- change to pg_standby to accept the parameter and use %r rather than -k
parameter

4. Add an option to pg_standby to have it complete all outstanding
archive files after it has been triggered, minimising data loss at the
slight expense of cut-over time

5. log_restartpoint option in recovery.conf
LOG each restorepoint, so can understand whether restartable recovery
will be effective or not

6. refactor recovery.conf so that it uses a GUC-like parser

7. refactor all xlog _desc routines into one module, so these can be
more easily used by xlogviewer utility

8. get xlogviewer utility a freshen-up so it can be part of main
release, possibly including further refactoring of xlog.c

9. Another round of doc updates to highlight the use of pg_standby and
Koichi-san's work. I think Doug Knight might have some additional code
examples to include as well, from previous discussions.

10. Changes to ensure WAL-avoiding operations and hot backups cannot be
executed simultaneously. One of these two options, ISTM:

a) Following a change to archive_command while server is running. Log
the xid of the WAL-avoiding operations when they start and have
pg_start_backup() wait for those xids to complete before continuing.

b) Introduce a new parameter, archive_mode = on | off that can only be
set at server start. If archive_mode = on then XLogArchivingActive();
archiving only takes place when archive_command is not ''. This allows
archive_command to be changed while server running, yet without any
danger from WAL-avoiding operations.

[7 & 8 would be complete by about 5-6 weeks from now. Others much
earlier]

Comments?

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Simon Riggs" <simon(at)2ndquadrant(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Minor changes to Recovery related code
Date: 2007-03-30 20:34:23
Message-ID: 23242.1175286863@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> I'd like to make the following changes to recovery related code over the
> next few days/weeks. If anybody insists I do this by freeze or not at
> all, then I'll submit patches for 1,3,4,5,10 before Saturday night. I'd
> rather take a bit more time and do this in one drop and there are some
> code dependencies between these changes and other patches from
> Koichi-san and myself.

Well, I've got a proposal for a pg_proc change that I haven't even started
coding yet, so personally I won't hold you to having the patch submitted
as long as the design is agreed to before feature freeze. However:

> 2. pg_stop_backup() should wait until all archive files are safely
> archived before returning

Not sure I agree with that one. If it fails, you can't tell whether the
action is done and it failed while waiting for the archiver, or if you
need to redo it.

> 6. refactor recovery.conf so that it uses a GUC-like parser

I would suggest that after feature freeze is not the time for code
beautification efforts like this. (I rather doubt that it's worth doing
at all actually, but definitely not now.)

> 7. refactor all xlog _desc routines into one module, so these can be
> more easily used by xlogviewer utility

Even more so.

> 8. get xlogviewer utility a freshen-up so it can be part of main
> release, possibly including further refactoring of xlog.c

This is not happening for 8.3, either.

> 10. Changes to ensure WAL-avoiding operations and hot backups cannot be
> executed simultaneously. One of these two options, ISTM:
> b) Introduce a new parameter, archive_mode = on | off that can only be
> set at server start. If archive_mode = on then XLogArchivingActive();
> archiving only takes place when archive_command is not ''. This allows
> archive_command to be changed while server running, yet without any
> danger from WAL-avoiding operations.

I think I'd go with (b) since a lot of people felt it should've been
like that from the beginning, and found the magic "empty string"
behavior confusing.

> [7 & 8 would be complete by about 5-6 weeks from now. Others much
> earlier]

We are hoping to go beta in less time than that. While I'm willing to
cut a little slack, anything you can't submit before about mid-April
is not going to make it into 8.3.

regards, tom lane


From: "Simon Riggs" <simon(at)2ndquadrant(dot)com>
To: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Minor changes to Recovery related code
Date: 2007-03-30 21:34:45
Message-ID: 1175290485.4386.848.camel@silverbirch.site
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > I'd like to make the following changes to recovery related code over the
> > next few days/weeks. If anybody insists I do this by freeze or not at
> > all, then I'll submit patches for 1,3,4,5,10 before Saturday night. I'd
> > rather take a bit more time and do this in one drop and there are some
> > code dependencies between these changes and other patches from
> > Koichi-san and myself.
>
> Well, I've got a proposal for a pg_proc change that I haven't even started
> coding yet, so personally I won't hold you to having the patch submitted
> as long as the design is agreed to before feature freeze. However:

Cool

> > 2. pg_stop_backup() should wait until all archive files are safely
> > archived before returning
>
> Not sure I agree with that one. If it fails, you can't tell whether the
> action is done and it failed while waiting for the archiver, or if you
> need to redo it.

There's a slight delay between pg_stop_backup() completing and the
archiver doing its stuff. Currently if somebody does a -m fast straight
after the pg_stop_backup() the backup may be unusable.

We need a way to plug that small hole.

I suggest that pg_stop_backup() polls once per second until
pg_xlog/archive_status/LOG.ready disappears, in which case it ends
successfully. If it does this for more than 60 seconds it ends
successfully but produces a WARNING.

> > 6. refactor recovery.conf so that it uses a GUC-like parser
>
> I would suggest that after feature freeze is not the time for code
> beautification efforts like this. (I rather doubt that it's worth doing
> at all actually, but definitely not now.)

OK

> > 7. refactor all xlog _desc routines into one module, so these can be
> > more easily used by xlogviewer utility
>
> Even more so.

OK

> > 8. get xlogviewer utility a freshen-up so it can be part of main
> > release, possibly including further refactoring of xlog.c
>
> This is not happening for 8.3, either.

OK

> > 10. Changes to ensure WAL-avoiding operations and hot backups cannot be
> > executed simultaneously. One of these two options, ISTM:
> > b) Introduce a new parameter, archive_mode = on | off that can only be
> > set at server start. If archive_mode = on then XLogArchivingActive();
> > archiving only takes place when archive_command is not ''. This allows
> > archive_command to be changed while server running, yet without any
> > danger from WAL-avoiding operations.
>
> I think I'd go with (b) since a lot of people felt it should've been
> like that from the beginning, and found the magic "empty string"
> behavior confusing.

OK

> We are hoping to go beta in less time than that. While I'm willing to
> cut a little slack, anything you can't submit before about mid-April
> is not going to make it into 8.3.

No probs, you struck a line through the more time consuming items. :-)

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com


From: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Minor changes to Recovery related code
Date: 2007-03-30 22:51:36
Message-ID: 460D9478.30505@phlo.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

Simon Riggs wrote:
> On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
>> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
>>> 2. pg_stop_backup() should wait until all archive files are safely
>>> archived before returning
>> Not sure I agree with that one. If it fails, you can't tell whether the
>> action is done and it failed while waiting for the archiver, or if you
>> need to redo it.
>
> There's a slight delay between pg_stop_backup() completing and the
> archiver doing its stuff. Currently if somebody does a -m fast straight
> after the pg_stop_backup() the backup may be unusable.
>
> We need a way to plug that small hole.
>
> I suggest that pg_stop_backup() polls once per second until
> pg_xlog/archive_status/LOG.ready disappears, in which case it ends
> successfully. If it does this for more than 60 seconds it ends
> successfully but produces a WARNING.

I fear that ending sucessfully despite having not archived all wals
will make this feature less worthwile. If a dba knows what he is
doing, he can code a perfectly safe backup script using 8.2 too.
He'll just have to check the current wal position after pg_stop_backup(),
(There is a function for that, right?), and wait until the corresponding
wal was archived.

In realitly, however, I feare that most people will just create a script
that does 'echo "select pg_stop_backup | psql"' or something similar.
If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
the return value of pgsql. I believe that those are the people who would
really benefit from a pg_stop_backup() that waits for archiving to complete.
But they probably won't check for WARNINGs.

Maybe doing it the other way round would be an option?
pg_stop_backup() could wait for the archiver to complete forever, but
spit out a warning every 60 seconds or so "WARNING: Still waiting
for wal archiving of wal ??? to complete". If someone really wants
a 60-second timeout, he can just use statement_timeout.

Anyway, just my 0.02 eurocents, maybe I'm totally mistaken about the
postgresql dba's out there...

greetings, Florian Pflug


From: Jim Nasby <decibel(at)decibel(dot)org>
To: Florian G(dot) Pflug <fgp(at)phlo(dot)org>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Minor changes to Recovery related code
Date: 2007-04-13 00:58:03
Message-ID: A2AAD09F-B22D-40F7-9F6A-C15447FF4A91@decibel.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

On Mar 30, 2007, at 5:51 PM, Florian G. Pflug wrote:
> In realitly, however, I feare that most people will just create a
> script
> that does 'echo "select pg_stop_backup | psql"' or something similar.
> If they're a bit more carefull, they will enable ON_ERROR_STOP, and
> check
> the return value of pgsql. I believe that those are the people who
> would
> really benefit from a pg_stop_backup() that waits for archiving to
> complete.
> But they probably won't check for WARNINGs.
>
> Maybe doing it the other way round would be an option?
> pg_stop_backup() could wait for the archiver to complete forever, but
> spit out a warning every 60 seconds or so "WARNING: Still waiting
> for wal archiving of wal ??? to complete". If someone really wants
> a 60-second timeout, he can just use statement_timeout.

I agree; people are far more likely to take the lazy way out, so it'd
be good to protect against that. Esp. since statement_timeout
provides a means to get the other behavior.
--
Jim Nasby jim(at)nasby(dot)net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)


From: "Simon Riggs" <simon(at)2ndquadrant(dot)com>
To: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>
Cc: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Minor changes to Recovery related code
Date: 2007-06-07 20:53:33
Message-ID: 1181249613.26886.271.camel@silverbirch.site
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
> Simon Riggs wrote:
> > On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
> >> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> >>> 2. pg_stop_backup() should wait until all archive files are safely
> >>> archived before returning
> >> Not sure I agree with that one. If it fails, you can't tell whether the
> >> action is done and it failed while waiting for the archiver, or if you
> >> need to redo it.
> >
> > There's a slight delay between pg_stop_backup() completing and the
> > archiver doing its stuff. Currently if somebody does a -m fast straight
> > after the pg_stop_backup() the backup may be unusable.
> >
> > We need a way to plug that small hole.
> >
> > I suggest that pg_stop_backup() polls once per second until
> > pg_xlog/archive_status/LOG.ready disappears, in which case it ends
> > successfully. If it does this for more than 60 seconds it ends
> > successfully but produces a WARNING.
>
> I fear that ending sucessfully despite having not archived all wals
> will make this feature less worthwile. If a dba knows what he is
> doing, he can code a perfectly safe backup script using 8.2 too.
> He'll just have to check the current wal position after pg_stop_backup(),
> (There is a function for that, right?), and wait until the corresponding
> wal was archived.
>
> In realitly, however, I feare that most people will just create a script
> that does 'echo "select pg_stop_backup | psql"' or something similar.
> If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
> the return value of pgsql. I believe that those are the people who would
> really benefit from a pg_stop_backup() that waits for archiving to complete.
> But they probably won't check for WARNINGs.
>
> Maybe doing it the other way round would be an option?
> pg_stop_backup() could wait for the archiver to complete forever, but
> spit out a warning every 60 seconds or so "WARNING: Still waiting
> for wal archiving of wal ??? to complete". If someone really wants
> a 60-second timeout, he can just use statement_timeout.

I've just come up against this problem again, so I think it is a must
fix for this release. Other problems exist also, mentioned on separate
threads.

We have a number of problems surrounding pg_stop_backup/shutdown:

1. pg_stop_backup() currently returns before the WAL file containing the
last change is correctly archived. That is a small hole, but one that is
exposed when people write test scripts that immediately shutdown the
database after issuing pg_stop_backup(). It doesn't make much sense to
shutdown immediately after a hot backup, but it should still work
sensibly.

2. We've also had problems caused by making the archiver wait until all
WAL files are archived. If there is a backlog for some reason and the
DBA issues a restart (i.e. stop and immediate restart) then making the
archiver loop while it tries (and possibly fails) to archive all files
would cause an outage. Avoiding this is why we do the current
get-out-fast approach.
There are some sub scenarios:
a) there is a backlog of WAL files, but no error has occurred on the
*last* file (we might have just fixed a problem).
b) there is a backlog of WAL files, but an error is causing a retry of
the last file.

My proposal is for us to record somewhere other than the logs that a
failure to archive has occurred and is being retried. Failure to archive
will be recorded in the archive_status directory as an additional file
called archive_error, which will be deleted in the case of archive
success and created in the case of archive error. This maintains
archiver's lack of attachment to shared memory and general simplicity of
design.

- pg_stop_backup() will wait until the WAL file that ends the backup is
safely archived, even if a failure to archive occurs. This is a change
to current behaviour, but since it implements the originally *expected*
behaviour IMHO it should be the default.

- new function: pg_stop_backup_nowait() return immediately without
waiting for archive, the same as the current pg_stop_backup()

- new function: pg_stop_backup_wait(int seconds) wait until either an
archival fails or the ending WAL file is archived, with a max wait as
specified. wait=0 means wait until archive errors are resolved.

Alternatives?

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com


From: "Simon Riggs" <simon(at)2ndquadrant(dot)com>
To: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>
Cc: "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us>, <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Minor changes to Recovery related code
Date: 2007-06-08 20:10:08
Message-ID: 1181333408.3643.39.camel@silverbirch.site
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:

> We have a number of problems surrounding pg_stop_backup/shutdown:

I'll start coding up my proposals, in the absence of alternate views.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com


From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Minor changes to Recovery related code
Date: 2007-09-27 05:23:07
Message-ID: 200709270523.l8R5N7621573@momjian.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches


This has been saved for the 8.4 release:

http://momjian.postgresql.org/cgi-bin/pgpatches_hold

---------------------------------------------------------------------------

Simon Riggs wrote:
> On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
> > Simon Riggs wrote:
> > > On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
> > >> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > >>> 2. pg_stop_backup() should wait until all archive files are safely
> > >>> archived before returning
> > >> Not sure I agree with that one. If it fails, you can't tell whether the
> > >> action is done and it failed while waiting for the archiver, or if you
> > >> need to redo it.
> > >
> > > There's a slight delay between pg_stop_backup() completing and the
> > > archiver doing its stuff. Currently if somebody does a -m fast straight
> > > after the pg_stop_backup() the backup may be unusable.
> > >
> > > We need a way to plug that small hole.
> > >
> > > I suggest that pg_stop_backup() polls once per second until
> > > pg_xlog/archive_status/LOG.ready disappears, in which case it ends
> > > successfully. If it does this for more than 60 seconds it ends
> > > successfully but produces a WARNING.
> >
> > I fear that ending sucessfully despite having not archived all wals
> > will make this feature less worthwile. If a dba knows what he is
> > doing, he can code a perfectly safe backup script using 8.2 too.
> > He'll just have to check the current wal position after pg_stop_backup(),
> > (There is a function for that, right?), and wait until the corresponding
> > wal was archived.
> >
> > In realitly, however, I feare that most people will just create a script
> > that does 'echo "select pg_stop_backup | psql"' or something similar.
> > If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
> > the return value of pgsql. I believe that those are the people who would
> > really benefit from a pg_stop_backup() that waits for archiving to complete.
> > But they probably won't check for WARNINGs.
> >
> > Maybe doing it the other way round would be an option?
> > pg_stop_backup() could wait for the archiver to complete forever, but
> > spit out a warning every 60 seconds or so "WARNING: Still waiting
> > for wal archiving of wal ??? to complete". If someone really wants
> > a 60-second timeout, he can just use statement_timeout.
>
> I've just come up against this problem again, so I think it is a must
> fix for this release. Other problems exist also, mentioned on separate
> threads.
>
> We have a number of problems surrounding pg_stop_backup/shutdown:
>
> 1. pg_stop_backup() currently returns before the WAL file containing the
> last change is correctly archived. That is a small hole, but one that is
> exposed when people write test scripts that immediately shutdown the
> database after issuing pg_stop_backup(). It doesn't make much sense to
> shutdown immediately after a hot backup, but it should still work
> sensibly.
>
> 2. We've also had problems caused by making the archiver wait until all
> WAL files are archived. If there is a backlog for some reason and the
> DBA issues a restart (i.e. stop and immediate restart) then making the
> archiver loop while it tries (and possibly fails) to archive all files
> would cause an outage. Avoiding this is why we do the current
> get-out-fast approach.
> There are some sub scenarios:
> a) there is a backlog of WAL files, but no error has occurred on the
> *last* file (we might have just fixed a problem).
> b) there is a backlog of WAL files, but an error is causing a retry of
> the last file.
>
> My proposal is for us to record somewhere other than the logs that a
> failure to archive has occurred and is being retried. Failure to archive
> will be recorded in the archive_status directory as an additional file
> called archive_error, which will be deleted in the case of archive
> success and created in the case of archive error. This maintains
> archiver's lack of attachment to shared memory and general simplicity of
> design.
>
> - pg_stop_backup() will wait until the WAL file that ends the backup is
> safely archived, even if a failure to archive occurs. This is a change
> to current behaviour, but since it implements the originally *expected*
> behaviour IMHO it should be the default.
>
> - new function: pg_stop_backup_nowait() return immediately without
> waiting for archive, the same as the current pg_stop_backup()
>
> - new function: pg_stop_backup_wait(int seconds) wait until either an
> archival fails or the ending WAL file is archived, with a max wait as
> specified. wait=0 means wait until archive errors are resolved.
>
> Alternatives?
>
> --
> Simon Riggs
> EnterpriseDB http://www.enterprisedb.com
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +


From: Simon Riggs <simon(at)2ndquadrant(dot)com>
To: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>, Bruce Momjian <bruce(at)momjian(dot)us>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: Minor changes to Recovery related code
Date: 2008-03-27 17:34:25
Message-ID: 1206639265.4285.1436.camel@ebony.site
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

Follow-up during March 2008 CommitFest

On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
> On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
> > Simon Riggs wrote:
> > > On Fri, 2007-03-30 at 16:34 -0400, Tom Lane wrote:
> > >> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > >>> 2. pg_stop_backup() should wait until all archive files are safely
> > >>> archived before returning
> > >> Not sure I agree with that one. If it fails, you can't tell whether the
> > >> action is done and it failed while waiting for the archiver, or if you
> > >> need to redo it.
> > >
> > > There's a slight delay between pg_stop_backup() completing and the
> > > archiver doing its stuff. Currently if somebody does a -m fast straight
> > > after the pg_stop_backup() the backup may be unusable.
> > >
> > > We need a way to plug that small hole.
> > >
> > > I suggest that pg_stop_backup() polls once per second until
> > > pg_xlog/archive_status/LOG.ready disappears, in which case it ends
> > > successfully. If it does this for more than 60 seconds it ends
> > > successfully but produces a WARNING.
> >
> > I fear that ending sucessfully despite having not archived all wals
> > will make this feature less worthwile. If a dba knows what he is
> > doing, he can code a perfectly safe backup script using 8.2 too.
> > He'll just have to check the current wal position after pg_stop_backup(),
> > (There is a function for that, right?), and wait until the corresponding
> > wal was archived.
> >
> > In realitly, however, I feare that most people will just create a script
> > that does 'echo "select pg_stop_backup | psql"' or something similar.
> > If they're a bit more carefull, they will enable ON_ERROR_STOP, and check
> > the return value of pgsql. I believe that those are the people who would
> > really benefit from a pg_stop_backup() that waits for archiving to complete.
> > But they probably won't check for WARNINGs.
> >
> > Maybe doing it the other way round would be an option?
> > pg_stop_backup() could wait for the archiver to complete forever, but
> > spit out a warning every 60 seconds or so "WARNING: Still waiting
> > for wal archiving of wal ??? to complete". If someone really wants
> > a 60-second timeout, he can just use statement_timeout.
>
> I've just come up against this problem again, so I think it is a must
> fix for this release. Other problems exist also, mentioned on separate
> threads.
>
> We have a number of problems surrounding pg_stop_backup/shutdown:
>
> 1. pg_stop_backup() currently returns before the WAL file containing the
> last change is correctly archived. That is a small hole, but one that is
> exposed when people write test scripts that immediately shutdown the
> database after issuing pg_stop_backup(). It doesn't make much sense to
> shutdown immediately after a hot backup, but it should still work
> sensibly.
>
> 2. We've also had problems caused by making the archiver wait until all
> WAL files are archived. If there is a backlog for some reason and the
> DBA issues a restart (i.e. stop and immediate restart) then making the
> archiver loop while it tries (and possibly fails) to archive all files
> would cause an outage. Avoiding this is why we do the current
> get-out-fast approach.
> There are some sub scenarios:
> a) there is a backlog of WAL files, but no error has occurred on the
> *last* file (we might have just fixed a problem).
> b) there is a backlog of WAL files, but an error is causing a retry of
> the last file.
>
> My proposal is for us to record somewhere other than the logs that a
> failure to archive has occurred and is being retried. Failure to archive
> will be recorded in the archive_status directory as an additional file
> called archive_error, which will be deleted in the case of archive
> success and created in the case of archive error. This maintains
> archiver's lack of attachment to shared memory and general simplicity of
> design.
>
> - pg_stop_backup() will wait until the WAL file that ends the backup is
> safely archived, even if a failure to archive occurs. This is a change
> to current behaviour, but since it implements the originally *expected*
> behaviour IMHO it should be the default.

The most straightforward thing is to make pg_stop_backup() wait as
described above.

If people want it to timeout, they can use a statement_timeout as
suggested by Florian.

This can be implemented by having the function poll in an infinite loop
for archive_status/LOG.done. Also, as Florian suggests, we should have
it output a WARNING message every 60 seconds.

I'll write a patch now.

> - new function: pg_stop_backup_nowait() return immediately without
> waiting for archive, the same as the current pg_stop_backup()
>
> - new function: pg_stop_backup_wait(int seconds) wait until either an
> archival fails or the ending WAL file is archived, with a max wait as
> specified. wait=0 means wait until archive errors are resolved.

So I won't do these.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk


From: Simon Riggs <simon(at)2ndquadrant(dot)com>
To: Bruce Momjian <bruce(at)momjian(dot)us>, pgsql-patches(at)postgresql(dot)org
Cc: "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Subject: Re: [HACKERS] Minor changes to Recovery related code
Date: 2008-04-01 16:17:34
Message-ID: 1207066654.4238.38.camel@ebony.site
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches

On Thu, 2008-03-27 at 17:34 +0000, Simon Riggs wrote:
> Follow-up during March 2008 CommitFest
>
> On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
> > On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:

> > - pg_stop_backup() will wait until the WAL file that ends the backup is
> > safely archived, even if a failure to archive occurs. This is a change
> > to current behaviour, but since it implements the originally *expected*
> > behaviour IMHO it should be the default.
>
> The most straightforward thing is to make pg_stop_backup() wait as
> described above.
>
> If people want it to timeout, they can use a statement_timeout as
> suggested by Florian.
>
> This can be implemented by having the function poll in an infinite loop
> for archive_status/LOG.done. Also, as Florian suggests, we should have
> it output a WARNING message every 60 seconds.
>
> I'll write a patch now.

Patch to implement waiting pg_stop_backup().

Tested and ready to apply, includes docs.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk

Attachment Content-Type Size
stop_backup_wait.v1.patch text/x-patch 7.6 KB

From: Bruce Momjian <bruce(at)momjian(dot)us>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: pgsql-patches(at)postgresql(dot)org, "Florian G(dot) Pflug" <fgp(at)phlo(dot)org>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Subject: Re: [HACKERS] Minor changes to Recovery related code
Date: 2008-04-05 01:34:58
Message-ID: 200804050134.m351YwM00633@momjian.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers pgsql-patches


Nice, applied. I only modified some of the documentation wording.

I was a little worried that statement_timeout might cancel
pg_stop_backup() _before_ it had gotten to waiting for the archive logs
to be saved, but based on what little code there is before that block, I
think we are OK.

---------------------------------------------------------------------------

Simon Riggs wrote:
> On Thu, 2008-03-27 at 17:34 +0000, Simon Riggs wrote:
> > Follow-up during March 2008 CommitFest
> >
> > On Thu, 2007-06-07 at 21:53 +0100, Simon Riggs wrote:
> > > On Sat, 2007-03-31 at 00:51 +0200, Florian G. Pflug wrote:
>
> > > - pg_stop_backup() will wait until the WAL file that ends the backup is
> > > safely archived, even if a failure to archive occurs. This is a change
> > > to current behaviour, but since it implements the originally *expected*
> > > behaviour IMHO it should be the default.
> >
> > The most straightforward thing is to make pg_stop_backup() wait as
> > described above.
> >
> > If people want it to timeout, they can use a statement_timeout as
> > suggested by Florian.
> >
> > This can be implemented by having the function poll in an infinite loop
> > for archive_status/LOG.done. Also, as Florian suggests, we should have
> > it output a WARNING message every 60 seconds.
> >
> > I'll write a patch now.
>
> Patch to implement waiting pg_stop_backup().
>
> Tested and ready to apply, includes docs.
>
> --
> Simon Riggs
> 2ndQuadrant http://www.2ndQuadrant.com
>
> PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk

[ Attachment, skipping... ]

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Attachment Content-Type Size
/rtmp/diff text/x-diff 7.5 KB