Re: Corner case in xlog stuff: what happens exactly at a

I'm noticing that if the current XLOG offset is exactly at a segment
boundary (ie, the last wal record just filled the segment) then the
various user-level functions return offsets that could be interpreted
as the start of the next segment, eg

regression=# select pg_switch_xlog();
pg_switch_xlog
----------------
0/14000000
(1 row)

regression=# select pg_xlogfile_from_wal_location(pg_switch_xlog());
pg_xlogfile_from_wal_location
-------------------------------
000000010000000000000014
(1 row)

regression=#

Actually, the last segment file that needs to be archived in this
scenario is 0013; it's possible 0014 doesn't even exist yet.

Rather than expecting user-level scripts to get this corner case
right, I suggest that we ought to modify pg_stop_backup and friends
so that what they return is the last used byte address of WAL, not
the first unused byte address as now. Then, blindly extracting
the filename will give the right answer about which file to archive,
even in the boundary case.

Comments?

regards, tom lane

I wrote:
> Rather than expecting user-level scripts to get this corner case
> right, I suggest that we ought to modify pg_stop_backup and friends
> so that what they return is the last used byte address of WAL, not
> the first unused byte address as now. Then, blindly extracting
> the filename will give the right answer about which file to archive,
> even in the boundary case.

After further thought I desisted from that plan: changing the result
convention of existing functions like pg_stop_backup() will break any
existing archiving scripts that do get it right. Instead, we can put
the boundary-case logic into the new functions that extract a filename
from the WAL location string that the action functions return.

regards, tom lane

On Sat, 2006-08-05 at 23:59 -0400, Tom Lane wrote:
> I wrote:
> > Rather than expecting user-level scripts to get this corner case
> > right, I suggest that we ought to modify pg_stop_backup and friends
> > so that what they return is the last used byte address of WAL, not
> > the first unused byte address as now. Then, blindly extracting
> > the filename will give the right answer about which file to archive,
> > even in the boundary case.
>
> After further thought I desisted from that plan: changing the result
> convention of existing functions like pg_stop_backup() will break any
> existing archiving scripts that do get it right. Instead, we can put
> the boundary-case logic into the new functions that extract a filename
> from the WAL location string that the action functions return.

This is done right? Ping me back if there's anything more to add.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com

Simon Riggs <simon(at)2ndquadrant(dot)com> writes:
> On Sat, 2006-08-05 at 23:59 -0400, Tom Lane wrote:
>> After further thought I desisted from that plan: changing the result
>> convention of existing functions like pg_stop_backup() will break any
>> existing archiving scripts that do get it right. Instead, we can put
>> the boundary-case logic into the new functions that extract a filename
>> from the WAL location string that the action functions return.

> This is done right? Ping me back if there's anything more to add.

It's done unless someone wants to change what I did ... at least as
far as the code goes. We might need more explanation in backup.sgml
about proper use of this stuff.

regards, tom lane

On Sat, Aug 05, 2006 at 11:59:40PM -0400, Tom Lane wrote:
> I wrote:
> > Rather than expecting user-level scripts to get this corner case
> > right, I suggest that we ought to modify pg_stop_backup and friends
> > so that what they return is the last used byte address of WAL, not
> > the first unused byte address as now. Then, blindly extracting
> > the filename will give the right answer about which file to archive,
> > even in the boundary case.
>
> After further thought I desisted from that plan: changing the result
> convention of existing functions like pg_stop_backup() will break any
> existing archiving scripts that do get it right. Instead, we can put
> the boundary-case logic into the new functions that extract a filename
> from the WAL location string that the action functions return.

Should we ask users if anyone is currently doing this? It seems pretty
ugly to have most functions return the last used WAL byte with only
stop_backup returning the first unused byte. If nothing else we should
at least plan on depricating this, probably by having a
pg_stop_backup(boolean) that lets the user specify which behavior they
want, and eventually switching pg_stop_backup() to the new behavior.

Ultimately, I'd bet that the vast majority of scripts out there blindly
do the wrong thing today and the authors aren't even aware of the issue.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

"Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> Should we ask users if anyone is currently doing this? It seems pretty
> ugly to have most functions return the last used WAL byte with only
> stop_backup returning the first unused byte.

No, you misunderstood. All the functions that return WAL location
strings use the same convention. It's the two new functions that
convert a WAL location string to a filename that have the boundary-case
logic.

regards, tom lane