Re: xlog file naming

Isn't the naming of the xlog files slightly bogus?

We have the following sequence:

00000001000008D0000000FD
00000001000008D0000000FE
00000001000008D100000000

Ignoring that we skip FF for some obscure reason (*), these are
effectively supposed to be 64-bit numbers, chunked into 16MB pieces, so
shouldn't the naming be

00000001000008D0FD000000
00000001000008D0FE000000
00000001000008D100000000

Then a lot of other things would also start making more sense:

The backup file

00000001000008D1000000C9.00013758.backup

should really be

00000001000008D1C9013758.backup

(At least conceptually. It's debatable whether we'd want to change
that, since as it is, it's convenient to detect the preceding WAL file
name by cutting off the end. OTOH, it's safer to actually look into the
backup file for that information.)

The return value of pg_start_backup that currently looks something like

pg_start_backup
-----------------
8D1/C9013758

should really be

000008D1C9013758

(perhaps the timeline should be included?)

and similarly, log output and pg_controldata output like

Latest checkpoint location: 8D3/5A1BB578

should be

Latest checkpoint location: 000008D35A1BB578

Then all instances of xlog location would look the same.

Also, the documentation offers this example:

"For example, if the starting WAL file is 0000000100001234000055CD the
backup history file will be named something like
0000000100001234000055CD.007C9330.backup."

Both the WAL and the backup file names used here are actually
impossible, and are not helping to clarify the issue.

It seems to me that this is all uselessly complicated and confused.

(*) - That idea appears to come from the same aboriginal confusion about
WAL "files" vs "segments" vs WAL position. Unless we support WAL
segments of size 1 or 2 bytes, there shouldn't be any risk of
overflowing the segment counter.

PS2: While we're discussing the cleanup of xlog.c, someone daring could
replace XLogRecPtr by a plain uint64 and possibly save hundres of lines
of code and eliminate a lot of the above confusion.

On 12.09.2011 21:36, Peter Eisentraut wrote:
> PS2: While we're discussing the cleanup of xlog.c, someone daring could
> replace XLogRecPtr by a plain uint64 and possibly save hundres of lines
> of code and eliminate a lot of the above confusion.

One problem with that is that it would break binary
backwards-compatibility on little-endian systems.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Isn't the naming of the xlog files slightly bogus?

No doubt, but by now there's enough replication-ish third-party code
that knows about them that I'm afraid changing these conventions would
be much more painful than it's worth.

regards, tom lane

On Tue, Sep 13, 2011 at 3:36 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> The return value of pg_start_backup that currently looks something like
>
>  pg_start_backup
> -----------------
>  8D1/C9013758
>
> should really be
>
>  000008D1C9013758
>
> (perhaps the timeline should be included?)

This change might break some tools which depends on such a result format.
Instead of that, what about introducing something like pg_current_timeline()
function which literally returns the current timeline ID, and extending
pg_xlogfile_name() so that it accepts not only LSN but also the timeline?
This idea enables us to get the backup start WAL file name by executing
"SELECT pg_xlogfile_name(pg_current_timeline(), pg_start_backup());".

Furthermore, in the same way, we can get the backup end WAL file name or
current WAL file name from pg_stop_backup() or pg_current_xlog_location(),
respectively.

> and similarly, log output and pg_controldata output like
>
> Latest checkpoint location:           8D3/5A1BB578
>
> should be
>
> Latest checkpoint location:           000008D35A1BB578

LSN information is useful for at least debug purpose. So, what about leaving
LSN information as it is and appending the filename in the end-of-line
as follows?
backup_label represents the backup start location in this way.

Latest checkpoint location: 0/2000020 (file 0000000000000002)

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Sep 12, 2011 at 8:44 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Tue, Sep 13, 2011 at 3:36 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>> The return value of pg_start_backup that currently looks something like
>>
>>  pg_start_backup
>> -----------------
>>  8D1/C9013758
>>
>> should really be
>>
>>  000008D1C9013758
>>
>> (perhaps the timeline should be included?)
>
> This change might break some tools which depends on such a result format.
> Instead of that, what about introducing something like pg_current_timeline()
> function which literally returns the current timeline ID,

until here i found it could have some use

> and extending
> pg_xlogfile_name() so that it accepts not only LSN but also the timeline?
> This idea enables us to get the backup start WAL file name by executing
> "SELECT pg_xlogfile_name(pg_current_timeline(), pg_start_backup());".
>

we can do "SELECT pg_xlogfile_name(pg_start_backup())" now, so how is
that different?

> Furthermore, in the same way, we can get the backup end WAL file name or
> current WAL file name from pg_stop_backup() or pg_current_xlog_location(),
> respectively.
>

we can do that already too

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación

On Tue, Sep 13, 2011 at 12:18 PM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
>> and extending
>> pg_xlogfile_name() so that it accepts not only LSN but also the timeline?
>> This idea enables us to get the backup start WAL file name by executing
>> "SELECT pg_xlogfile_name(pg_current_timeline(), pg_start_backup());".
>
> we can do "SELECT pg_xlogfile_name(pg_start_backup())" now, so how is
> that different?

Yeah, no difference. My idea is obviously useless for pg_start_backup()..

What I had in mind was to use the extended pg_xlogfile_name() to calculate
the WAL filename from (for example) pg_controldata executed on the standby.
The timeline which pg_controldata displays might be different from the
current one in the master. So pg_xlogfile_name() which always use the current
timeline might return a wrong WAL filename. Allowing pg_xlogfile_name() to
use the specified timeline to calculate a filename would address that problem.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Are there any TODO items here?

---------------------------------------------------------------------------

On Mon, Sep 12, 2011 at 09:36:02PM +0300, Peter Eisentraut wrote:
> Isn't the naming of the xlog files slightly bogus?
>
> We have the following sequence:
>
> 00000001000008D0000000FD
> 00000001000008D0000000FE
> 00000001000008D100000000
>
> Ignoring that we skip FF for some obscure reason (*), these are
> effectively supposed to be 64-bit numbers, chunked into 16MB pieces, so
> shouldn't the naming be
>
> 00000001000008D0FD000000
> 00000001000008D0FE000000
> 00000001000008D100000000
>
> Then a lot of other things would also start making more sense:
>
> The backup file
>
> 00000001000008D1000000C9.00013758.backup
>
> should really be
>
> 00000001000008D1C9013758.backup
>
> (At least conceptually. It's debatable whether we'd want to change
> that, since as it is, it's convenient to detect the preceding WAL file
> name by cutting off the end. OTOH, it's safer to actually look into the
> backup file for that information.)
>
> The return value of pg_start_backup that currently looks something like
>
> pg_start_backup
> -----------------
> 8D1/C9013758
>
> should really be
>
> 000008D1C9013758
>
> (perhaps the timeline should be included?)
>
> and similarly, log output and pg_controldata output like
>
> Latest checkpoint location: 8D3/5A1BB578
>
> should be
>
> Latest checkpoint location: 000008D35A1BB578
>
> Then all instances of xlog location would look the same.
>
> Also, the documentation offers this example:
>
> "For example, if the starting WAL file is 0000000100001234000055CD the
> backup history file will be named something like
> 0000000100001234000055CD.007C9330.backup."
>
> Both the WAL and the backup file names used here are actually
> impossible, and are not helping to clarify the issue.
>
> It seems to me that this is all uselessly complicated and confused.
>
>
> (*) - That idea appears to come from the same aboriginal confusion about
> WAL "files" vs "segments" vs WAL position. Unless we support WAL
> segments of size 1 or 2 bytes, there shouldn't be any risk of
> overflowing the segment counter.
>
>
> PS2: While we're discussing the cleanup of xlog.c, someone daring could
> replace XLogRecPtr by a plain uint64 and possibly save hundres of lines
> of code and eliminate a lot of the above confusion.
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Wed, Aug 15, 2012 at 8:43 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Are there any TODO items here?

It's possible there's something we want to change here, but it's far
from obvious what that thing is. Our WAL file handling is
ridiculously hard to understand, but the problem with changing it is
that there will then be two things people have to understand, and a
lot of tools that have to be revamped. It isn't clear that it's worth
going through that kind of pain for a minor improvement in clarity.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, 2012-08-21 at 12:01 -0400, Robert Haas wrote:
> On Wed, Aug 15, 2012 at 8:43 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > Are there any TODO items here?
>
> It's possible there's something we want to change here, but it's far
> from obvious what that thing is. Our WAL file handling is
> ridiculously hard to understand, but the problem with changing it is
> that there will then be two things people have to understand, and a
> lot of tools that have to be revamped. It isn't clear that it's worth
> going through that kind of pain for a minor improvement in clarity.

The idea was that since we already broke some tools, possibly silently
(...FF files that they previously skipped), a more radical renaming
might break those tools more obviously, and make some other things
simpler/easier down the road.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On Tue, 2012-08-21 at 12:01 -0400, Robert Haas wrote:
>> It's possible there's something we want to change here, but it's far
>> from obvious what that thing is. Our WAL file handling is
>> ridiculously hard to understand, but the problem with changing it is
>> that there will then be two things people have to understand, and a
>> lot of tools that have to be revamped. It isn't clear that it's worth
>> going through that kind of pain for a minor improvement in clarity.

> The idea was that since we already broke some tools, possibly silently
> (...FF files that they previously skipped), a more radical renaming
> might break those tools more obviously, and make some other things
> simpler/easier down the road.

I think we already had that discussion, and the consensus was that
we did not want to break WAL-related tools unnecessarily. If there
were a high probability that the FF change will actually break tools in
practice, the conclusion might have been different; but nobody believes
that there is much risk there.

regards, tom lane