Re: Reporting WAL file containing checkpoint's REDO record in pg_controldata's result

Hi,

I'd like to propose to change pg_controldata so that it reports the name
of WAL file containing the latest checkpoint's REDO record, as follows:

$ pg_controldata $PGDATA
...
Latest checkpoint's REDO location: 0/16D6ACC (file
000000010000000000000001)
Latest checkpoint's TimeLineID: 1
...

This simplifies very much the way to calculate the archive file cutoff point
because the reported WAL file is just cutoff point itself. If the file name is
not reported, we have to calculate the cutoff point from the reported
location and timeline, which is complicated calculation. We can use
pg_xlogfile_name function to calculate that, but it cannot be executed in
the standby. Another problem is that pg_xlogfile_name always uses
current timeline for the calculation, so if the reported timeline is not
the same as current one, pg_xlogfile_name cannot return the correct WAL
file name. Making pg_controldata report that WAL file name gets rid of
such a complexity.

You may think that archive_cleanup_command is usable for that purpose.
That's true. But it's not usable simply for the case where there are more
than one standby servers. In this case, the archive file cutoff point needs
to be calculated from each standby's REDO location and timeline.

Attached patch changes pg_controldata as above. Thought?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>
> We can use
> pg_xlogfile_name function to calculate that, but it cannot be executed in
> the standby. Another problem is that pg_xlogfile_name always uses
> current timeline for the calculation, so if the reported timeline is not
> the same as current one, pg_xlogfile_name cannot return the correct WAL
> file name. Making pg_controldata report that WAL file name gets rid of
> such a complexity.
>

i would think that pg_xlogfile_name() is not allowed in the standby
because ThisTimelineId is not very well defined in recovery but if you
extend pg_xlogfile_name() to also receive a timelineid as you
suggested in [1] then that version of the function could be allowed in
the standby.
or there is something else i'm missing?

is that enough for you to solve your problem?

[1] http://archives.postgresql.org/message-id/CAHGQGwHWqjGeksmP2OterU8P0SJ6X7yPQyH5qqEqcBQSXaAXaw@mail.gmail.com

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación

On Fri, Mar 23, 2012 at 05:06, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> Hi,
>
> I'd like to propose to change pg_controldata so that it reports the name
> of WAL file containing the latest checkpoint's REDO record, as follows:
>
>    $ pg_controldata $PGDATA
>    ...
>    Latest checkpoint's REDO location:    0/16D6ACC (file
> 000000010000000000000001)
>    Latest checkpoint's TimeLineID:       1
>    ...
>
> This simplifies very much the way to calculate the archive file cutoff point
> because the reported WAL file is just cutoff point itself. If the file name is
> not reported, we have to calculate the cutoff point from the reported
> location and timeline, which is complicated calculation. We can use
> pg_xlogfile_name function to calculate that, but it cannot be executed in
> the standby. Another problem is that pg_xlogfile_name always uses
> current timeline for the calculation, so if the reported timeline is not
> the same as current one, pg_xlogfile_name cannot return the correct WAL
> file name. Making pg_controldata report that WAL file name gets rid of
> such a complexity.
>
> You may think that archive_cleanup_command is usable for that purpose.
> That's true. But it's not usable simply for  the case where there are more
> than one standby servers. In this case, the archive file cutoff point needs
> to be calculated from each standby's REDO location and timeline.
>
> Attached patch changes pg_controldata as above. Thought?

Might it be a good idea to put it on it's own row instead of changing
the format of an existing row, in order not to break scripts and
programs that are parsing the previous output?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Fri, Mar 23, 2012 at 2:06 PM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
> On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>
>> We can use
>> pg_xlogfile_name function to calculate that, but it cannot be executed in
>> the standby. Another problem is that pg_xlogfile_name always uses
>> current timeline for the calculation, so if the reported timeline is not
>> the same as current one, pg_xlogfile_name cannot return the correct WAL
>> file name. Making pg_controldata report that WAL file name gets rid of
>> such a complexity.
>>
>
> i would think that pg_xlogfile_name() is not allowed in the standby
> because ThisTimelineId is not very well defined in recovery but if you
> extend pg_xlogfile_name() to also receive a timelineid as you
> suggested in [1] then that version of the function could be allowed in
> the standby.
> or there is something else i'm missing?
>
> is that enough for you to solve your problem?

Yes, we can more easily calculate the cutoff point by using that extended
pg_xlogfile_name(). But if pg_controldata reports the WAL file name, we
can calculate the cutoff point without starting the server. So I think that
it's worth changing pg_controldata that way even if we extend
pg_xlogfile_name().

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Mar 23, 2012 at 10:51, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Mar 23, 2012 at 2:06 PM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
>> On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>>
>>> We can use
>>> pg_xlogfile_name function to calculate that, but it cannot be executed in
>>> the standby. Another problem is that pg_xlogfile_name always uses
>>> current timeline for the calculation, so if the reported timeline is not
>>> the same as current one, pg_xlogfile_name cannot return the correct WAL
>>> file name. Making pg_controldata report that WAL file name gets rid of
>>> such a complexity.
>>>
>>
>> i would think that pg_xlogfile_name() is not allowed in the standby
>> because ThisTimelineId is not very well defined in recovery but if you
>> extend pg_xlogfile_name() to also receive a timelineid as you
>> suggested in [1] then that version of the function could be allowed in
>> the standby.
>> or there is something else i'm missing?
>>
>> is that enough for you to solve your problem?
>
> Yes, we can more easily calculate the cutoff point by using that extended
> pg_xlogfile_name(). But if pg_controldata reports the WAL file name, we
> can calculate the cutoff point without starting the server. So I think that
> it's worth changing pg_controldata that way even if we extend
> pg_xlogfile_name().

+1 - I think they're both useful things, each on it's own.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> Might it be a good idea to put it on it's own row instead of changing
> the format of an existing row, in order not to break scripts and
> programs that are parsing the previous output?

Good idea! What row name should we use for the WAL file containing
REDO record? "Latest checkpoint's REDO file"?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> Might it be a good idea to put it on it's own row instead of changing
>> the format of an existing row, in order not to break scripts and
>> programs that are parsing the previous output?
>
> Good idea! What row name should we use for the WAL file containing
> REDO record? "Latest checkpoint's REDO file"?

Sounds good to me. I like the idea, too. The status quo is an
unnecessary nuisance, so this will be a nice usability improvement.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> Might it be a good idea to put it on it's own row instead of changing
>>> the format of an existing row, in order not to break scripts and
>>> programs that are parsing the previous output?
>>
>> Good idea! What row name should we use for the WAL file containing
>> REDO record? "Latest checkpoint's REDO file"?
>
> Sounds good to me.  I like the idea, too.  The status quo is an
> unnecessary nuisance, so this will be a nice usability improvement.

Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into
the result of pg_controldata. I used the term "WAL segment" for the row name
instead of "file" because "WAL segment" is used in another row "Bytes per WAL
segment:". But better name?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> Might it be a good idea to put it on it's own row instead of changing
>>>> the format of an existing row, in order not to break scripts and
>>>> programs that are parsing the previous output?
>>>
>>> Good idea! What row name should we use for the WAL file containing
>>> REDO record? "Latest checkpoint's REDO file"?
>>
>> Sounds good to me.  I like the idea, too.  The status quo is an
>> unnecessary nuisance, so this will be a nice usability improvement.
>
> Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into
> the result of pg_controldata. I used the term "WAL segment" for the row name
> instead of "file" because "WAL segment" is used in another row "Bytes per WAL
> segment:". But better name?

s/segment/file/g?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Sat, Mar 24, 2012 at 1:49 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>>> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>>> Might it be a good idea to put it on it's own row instead of changing
>>>>> the format of an existing row, in order not to break scripts and
>>>>> programs that are parsing the previous output?
>>>>
>>>> Good idea! What row name should we use for the WAL file containing
>>>> REDO record? "Latest checkpoint's REDO file"?
>>>
>>> Sounds good to me.  I like the idea, too.  The status quo is an
>>> unnecessary nuisance, so this will be a nice usability improvement.
>>
>> Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into
>> the result of pg_controldata. I used the term "WAL segment" for the row name
>> instead of "file" because "WAL segment" is used in another row "Bytes per WAL
>> segment:". But better name?
>
> s/segment/file/g?

Yep, "file" might be more intuitive for a user than "segment". Attached is the
"file" version of the patch.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Mar 23, 2012 at 18:05, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Sat, Mar 24, 2012 at 1:49 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>>> On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>>>> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>>>> Might it be a good idea to put it on it's own row instead of changing
>>>>>> the format of an existing row, in order not to break scripts and
>>>>>> programs that are parsing the previous output?
>>>>>
>>>>> Good idea! What row name should we use for the WAL file containing
>>>>> REDO record? "Latest checkpoint's REDO file"?
>>>>
>>>> Sounds good to me.  I like the idea, too.  The status quo is an
>>>> unnecessary nuisance, so this will be a nice usability improvement.
>>>
>>> Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into
>>> the result of pg_controldata. I used the term "WAL segment" for the row name
>>> instead of "file" because "WAL segment" is used in another row "Bytes per WAL
>>> segment:". But better name?
>>
>> s/segment/file/g?
>
> Yep, "file" might be more intuitive for a user than "segment". Attached is the
> "file" version of the patch.

We're already using "file" to mean something different *internally*,
don't we? And since pg_controldata shows fairly internal information,
I'm not sure this is the best idea.

Maybe compromise and call it "segment file" - that is both easier to
understand than segment, and not actually using a term that means
something else...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> s/segment/file/g?
>>
>> Yep, "file" might be more intuitive for a user than "segment". Attached is the
>> "file" version of the patch.
>
> We're already using "file" to mean something different *internally*,
> don't we? And since pg_controldata shows fairly internal information,
> I'm not sure this is the best idea.
>
> Maybe compromise and call it "segment file" - that is both easier to
> understand than segment, and not actually using a term that means
> something else...

It's also kind of wordy. I think "file" is fine. There are a few
references to xlogid indicating a "file number", but the actual field
name is just xlogid. We also use the term "file" to mean the other
thing, as in XLOGfileslop, and I have a hard time believing anyone's
really going to get confused about what is meant here.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> s/segment/file/g?

>> We're already using "file" to mean something different *internally*,
>> don't we? And since pg_controldata shows fairly internal information,
>> I'm not sure this is the best idea.
>>
>> Maybe compromise and call it "segment file" - that is both easier to
>> understand than segment, and not actually using a term that means
>> something else...

> It's also kind of wordy. I think "file" is fine.

+1 for "file". I think the internal usage of "file" to mean "roughly
4GB worth of WAL" is going to go away soon anyway, as there won't be
much reason to worry about the concept once LSN arithmetic is 64-bit.

regards, tom lane

On Mon, Mar 26, 2012 at 11:18 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> s/segment/file/g?
>
>>> We're already using "file" to mean something different *internally*,
>>> don't we? And since pg_controldata shows fairly internal information,
>>> I'm not sure this is the best idea.
>>>
>>> Maybe compromise and call it "segment file" - that is both easier to
>>> understand than segment, and not actually using a term that means
>>> something else...
>
>> It's also kind of wordy.  I think "file" is fine.
>
> +1 for "file".  I think the internal usage of "file" to mean "roughly
> 4GB worth of WAL" is going to go away soon anyway, as there won't be
> much reason to worry about the concept once LSN arithmetic is 64-bit.

Agreed. This would mean that the following lots of log messages need to
be changed after 64-bit LSN will have been committed.

errmsg("could not fdatasync log file %u, segment %u: %m",
log, seg)));

Anyway, should I add this patch into the next CF? Or is anyone planning
to commit the patch for 9.2?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012:

> Anyway, should I add this patch into the next CF? Or is anyone planning
> to commit the patch for 9.2?

I think the correct thing to do here is add to next CF, and if some
committer has enough interest in getting it quickly it can be grabbed
from there and committed into 9.2.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:
>
> Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012:
>
>> Anyway, should I add this patch into the next CF? Or is anyone planning
>> to commit the patch for 9.2?
>
> I think the correct thing to do here is add to next CF, and if some
> committer has enough interest in getting it quickly it can be grabbed
> from there and committed into 9.2.

Yep! I've added it to next CF.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Mar 28, 2012 at 10:08 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera
> <alvherre(at)commandprompt(dot)com> wrote:
>>
>> Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012:
>>
>>> Anyway, should I add this patch into the next CF? Or is anyone planning
>>> to commit the patch for 9.2?
>>
>> I think the correct thing to do here is add to next CF, and if some
>> committer has enough interest in getting it quickly it can be grabbed
>> from there and committed into 9.2.
>
> Yep! I've added it to next CF.

Attached is the revised version of the patch.

Regards,

--
Fujii Masao

On 26.06.2012 18:06, Fujii Masao wrote:
> On Wed, Mar 28, 2012 at 10:08 AM, Fujii Masao<masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera
>> <alvherre(at)commandprompt(dot)com> wrote:
>>>
>>> Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012:
>>>
>>>> Anyway, should I add this patch into the next CF? Or is anyone planning
>>>> to commit the patch for 9.2?
>>>
>>> I think the correct thing to do here is add to next CF, and if some
>>> committer has enough interest in getting it quickly it can be grabbed
>>> from there and committed into 9.2.
>>
>> Yep! I've added it to next CF.
>
> Attached is the revised version of the patch.

Looks good to me, committed.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com