Re: pg_retainxlog for inclusion in 9.3?

So, it turns out the reason I got no feedback on this tool, was that I
forgot both to email about and to actually push the code to github :O
So this is actually code that's almost half a year old and that I was
supposed to submit for the first or second commitfest. Oops.

So, the tool and a README for it right now can be found at
https://github.com/mhagander/pg_retainxlog for the time being.

The idea behind the tool is to plug a hole in the case when
pg_receivexlog is used for log archiving, which is that you still need
a "blocking" archive_command in order to make sure that files aren't
recycled on the master. So for 9.2 you can do this with an
archive_command that checks if the file has appeared properly on the
slave - but that usually means you're back at requiring ssh
connectivity between the machines, for example. Even though this
information is actually avialable on the master...

This can also be of use to pure replication scenarios, where you don't
know how to tune wal_keep_segments, but using actual live feedback
instead of guessing.

When pg_retainxlog is used as an archive_command, it will check the
pg_stat_replication view instead of checking the slave. It will then
only return ok once the requested logfile has been replicated to the
slave. By default it will look for a replication client named
pg_receivexlog, but it supports overriding the query with anything -
so you can say things like "needs to have arrived on at least two
replication slaves before we consider it archived". Or if used instead
of wal_keep_segmnets, needs to have arrived at *all* replication
slaves.

Is this a tool that people would like to see included in the general
toolchain? If so, I'll reformat it to work in the general build
environment and submit it for the last commitfest.

(comments on the code itself are of course also welcome)

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 01/01/2013 04:10 PM, Magnus Hagander wrote:
> So, it turns out the reason I got no feedback on this tool, was that I
> forgot both to email about and to actually push the code to github :O
> So this is actually code that's almost half a year old and that I was
> supposed to submit for the first or second commitfest. Oops.
>
> So, the tool and a README for it right now can be found at
> https://github.com/mhagander/pg_retainxlog for the time being.
>
> The idea behind the tool is to plug a hole in the case when
> pg_receivexlog is used for log archiving, which is that you still need
> a "blocking" archive_command in order to make sure that files aren't
> recycled on the master. So for 9.2 you can do this with an
> archive_command that checks if the file has appeared properly on the
> slave - but that usually means you're back at requiring ssh
> connectivity between the machines, for example. Even though this
> information is actually avialable on the master...
>
> This can also be of use to pure replication scenarios, where you don't
> know how to tune wal_keep_segments, but using actual live feedback
> instead of guessing.
>
> When pg_retainxlog is used as an archive_command, it will check the
> pg_stat_replication view instead of checking the slave. It will then
> only return ok once the requested logfile has been replicated to the
> slave. By default it will look for a replication client named
> pg_receivexlog, but it supports overriding the query with anything -
> so you can say things like "needs to have arrived on at least two
> replication slaves before we consider it archived". Or if used instead
> of wal_keep_segmnets, needs to have arrived at *all* replication
> slaves.
>
> Is this a tool that people would like to see included in the general
> toolchain? If so, I'll reformat it to work in the general build
> environment and submit it for the last commitfest.
>

+1

------------------------
Hannu Krosing

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Is this a tool that people would like to see included in the general
> toolchain? If so, I'll reformat it to work in the general build
> environment and submit it for the last commitfest.

Please do.

Regards,
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> So, it turns out the reason I got no feedback on this tool, was that I
> forgot both to email about and to actually push the code to github :O
> So this is actually code that's almost half a year old and that I was
> supposed to submit for the first or second commitfest. Oops.
>
> So, the tool and a README for it right now can be found at
> https://github.com/mhagander/pg_retainxlog for the time being.
>
> The idea behind the tool is to plug a hole in the case when
> pg_receivexlog is used for log archiving, which is that you still need
> a "blocking" archive_command in order to make sure that files aren't
> recycled on the master. So for 9.2 you can do this with an
> archive_command that checks if the file has appeared properly on the
> slave - but that usually means you're back at requiring ssh
> connectivity between the machines, for example. Even though this
> information is actually avialable on the master...
>
> This can also be of use to pure replication scenarios, where you don't
> know how to tune wal_keep_segments, but using actual live feedback
> instead of guessing.
>
> When pg_retainxlog is used as an archive_command, it will check the
> pg_stat_replication view instead of checking the slave. It will then
> only return ok once the requested logfile has been replicated to the
> slave. By default it will look for a replication client named
> pg_receivexlog, but it supports overriding the query with anything -
> so you can say things like "needs to have arrived on at least two
> replication slaves before we consider it archived". Or if used instead
> of wal_keep_segmnets, needs to have arrived at *all* replication
> slaves.
>
> Is this a tool that people would like to see included in the general
> toolchain? If so, I'll reformat it to work in the general build
> environment and submit it for the last commitfest.
>
> (comments on the code itself are of course also welcome)

I would tend to vote for putting this in contrib rather than src/bin.
But apart from that I have no objection to the idea.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Jan 3, 2013 at 3:13 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> So, it turns out the reason I got no feedback on this tool, was that I
>> forgot both to email about and to actually push the code to github :O
>> So this is actually code that's almost half a year old and that I was
>> supposed to submit for the first or second commitfest. Oops.
>>
>> So, the tool and a README for it right now can be found at
>> https://github.com/mhagander/pg_retainxlog for the time being.
>>
>> The idea behind the tool is to plug a hole in the case when
>> pg_receivexlog is used for log archiving, which is that you still need
>> a "blocking" archive_command in order to make sure that files aren't
>> recycled on the master. So for 9.2 you can do this with an
>> archive_command that checks if the file has appeared properly on the
>> slave - but that usually means you're back at requiring ssh
>> connectivity between the machines, for example. Even though this
>> information is actually avialable on the master...
>>
>> This can also be of use to pure replication scenarios, where you don't
>> know how to tune wal_keep_segments, but using actual live feedback
>> instead of guessing.
>>
>> When pg_retainxlog is used as an archive_command, it will check the
>> pg_stat_replication view instead of checking the slave. It will then
>> only return ok once the requested logfile has been replicated to the
>> slave. By default it will look for a replication client named
>> pg_receivexlog, but it supports overriding the query with anything -
>> so you can say things like "needs to have arrived on at least two
>> replication slaves before we consider it archived". Or if used instead
>> of wal_keep_segmnets, needs to have arrived at *all* replication
>> slaves.
>>
>> Is this a tool that people would like to see included in the general
>> toolchain? If so, I'll reformat it to work in the general build
>> environment and submit it for the last commitfest.
>>
>> (comments on the code itself are of course also welcome)
>
> I would tend to vote for putting this in contrib rather than src/bin.
> But apart from that I have no objection to the idea.

Any particular reason? It goes pretty tightly together with
pg_receivexlog, which is why I'd prefer putting it alongside that one.
But if you have a good argument against it, I can change my mind :)

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> Any particular reason? It goes pretty tightly together with
> pg_receivexlog, which is why I'd prefer putting it alongside that one.
> But if you have a good argument against it, I can change my mind :)

Mostly that it seems like a hack, and I suspect we may come up with a
better way to do this in the future.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 1/3/13 12:30 PM, Robert Haas wrote:
> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> Any particular reason? It goes pretty tightly together with
>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
>> But if you have a good argument against it, I can change my mind :)
>
> Mostly that it seems like a hack, and I suspect we may come up with a
> better way to do this in the future.

It does seem like a hack. Couldn't this be implemented with a backend
switch instead?

Also, as a small practical matter, since this is a server-side program
(since it's being used as archive_command), we shouldn't put it into the
pg_basebackup directory, because that would blur the lines about what to
install where, in particular for the translations.

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Mostly that it seems like a hack, and I suspect we may come up with a
> better way to do this in the future.

Do you have the specs of such better way? Would it be a problem to have
both pg_retainxlog and the new way?

--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On 1/3/13 12:30 PM, Robert Haas wrote:
>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> Any particular reason? It goes pretty tightly together with
>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
>>> But if you have a good argument against it, I can change my mind :)
>>
>> Mostly that it seems like a hack, and I suspect we may come up with a
>> better way to do this in the future.
>
> It does seem like a hack. Couldn't this be implemented with a backend
> switch instead?

It definitely is a bit of a hack.

I assume by backend switch you mean guc, right? If so, no, not easily
so. Because it's the archiver process that does the deleting. And this
process does not have access to a "full backend interface", e.g. the
ability to run a query. We could make it look at the same data that's
currently shown in pg_stat_replicatoin through shared memory, but thta
would *only* work in the very most simple cases (e.g. a single
pg_receivexlog and no other replication). The ability to run a custom
SQL query is going to be necessary for anything a bit more advanced.

> Also, as a small practical matter, since this is a server-side program
> (since it's being used as archive_command), we shouldn't put it into the
> pg_basebackup directory, because that would blur the lines about what to
> install where, in particular for the translations.

Good argument. That along with the being a hack, and the comment from
Robert, means that maybe contrib/ is a better place for it, yes.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Tue, Jan 1, 2013 at 10:10 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> So, it turns out the reason I got no feedback on this tool, was that I
> forgot both to email about and to actually push the code to github :O
> So this is actually code that's almost half a year old and that I was
> supposed to submit for the first or second commitfest. Oops.
>
> So, the tool and a README for it right now can be found at
> https://github.com/mhagander/pg_retainxlog for the time being.
>
> The idea behind the tool is to plug a hole in the case when
> pg_receivexlog is used for log archiving, which is that you still need
> a "blocking" archive_command in order to make sure that files aren't
> recycled on the master. So for 9.2 you can do this with an
> archive_command that checks if the file has appeared properly on the
> slave - but that usually means you're back at requiring ssh
> connectivity between the machines, for example. Even though this
> information is actually avialable on the master...
>
> This can also be of use to pure replication scenarios, where you don't
> know how to tune wal_keep_segments, but using actual live feedback
> instead of guessing.
>
> When pg_retainxlog is used as an archive_command, it will check the
> pg_stat_replication view instead of checking the slave. It will then
> only return ok once the requested logfile has been replicated to the
> slave. By default it will look for a replication client named
> pg_receivexlog, but it supports overriding the query with anything -
> so you can say things like "needs to have arrived on at least two
> replication slaves before we consider it archived". Or if used instead
> of wal_keep_segmnets, needs to have arrived at *all* replication
> slaves.
>
> Is this a tool that people would like to see included in the general
> toolchain? If so, I'll reformat it to work in the general build
> environment and submit it for the last commitfest.
>
> (comments on the code itself are of course also welcome)
>
> --
> Magnus Hagander
> Me: http://www.hagander.net/
> Work: http://www.redpill-linpro.com/
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

+1 to this concept, however it may be implemented.

On Sat, Jan 5, 2013 at 3:11 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>> On 1/3/13 12:30 PM, Robert Haas wrote:
>>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> Any particular reason? It goes pretty tightly together with
>>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
>>>> But if you have a good argument against it, I can change my mind :)
>>>
>>> Mostly that it seems like a hack, and I suspect we may come up with a
>>> better way to do this in the future.
>>
>> It does seem like a hack. Couldn't this be implemented with a backend
>> switch instead?
>
> It definitely is a bit of a hack.
>
> I assume by backend switch you mean guc, right? If so, no, not easily
> so. Because it's the archiver process that does the deleting. And this
> process does not have access to a "full backend interface", e.g. the
> ability to run a query. We could make it look at the same data that's
> currently shown in pg_stat_replicatoin through shared memory, but thta
> would *only* work in the very most simple cases (e.g. a single
> pg_receivexlog and no other replication). The ability to run a custom
> SQL query is going to be necessary for anything a bit more advanced.
>
>
>> Also, as a small practical matter, since this is a server-side program
>> (since it's being used as archive_command), we shouldn't put it into the
>> pg_basebackup directory, because that would blur the lines about what to
>> install where, in particular for the translations.
>
> Good argument. That along with the being a hack, and the comment from
> Robert, means that maybe contrib/ is a better place for it, yes.

Here's a version for inclusion in /contrib.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Mostly that it seems like a hack, and I suspect we may come up with a
>> better way to do this in the future.
>
> Do you have the specs of such better way? Would it be a problem to have
> both pg_retainxlog and the new way?

Well, I think in the long term we are likely to want the master to
have some kind of ability to track the positions of its slaves, even
when they are disconnected. And, optionally, to retain the WAL that
they need, again even when they are disconnected. If such an ability
materializes, this will be moot (even as I think that pg_standby is
now largely moot, at least for new installations, now that we have
standby_mode=on).

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Jan 14, 2013 at 5:56 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> Mostly that it seems like a hack, and I suspect we may come up with a
>>> better way to do this in the future.
>>
>> Do you have the specs of such better way? Would it be a problem to have
>> both pg_retainxlog and the new way?
>
> Well, I think in the long term we are likely to want the master to
> have some kind of ability to track the positions of its slaves, even
> when they are disconnected. And, optionally, to retain the WAL that
> they need, again even when they are disconnected. If such an ability
> materializes, this will be moot (even as I think that pg_standby is
> now largely moot, at least for new installations, now that we have
> standby_mode=on).

I agree. But just as we had pg_standby for quite a while before we got
standby_mode=on, I believe we should have pg_retainxlog (or something
like it) until we have something more integrated.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Tue, Jan 15, 2013 at 2:51 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Mon, Jan 14, 2013 at 5:56 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Fri, Jan 4, 2013 at 4:55 PM, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
>>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>>> Mostly that it seems like a hack, and I suspect we may come up with a
>>>> better way to do this in the future.
>>>
>>> Do you have the specs of such better way? Would it be a problem to have
>>> both pg_retainxlog and the new way?
>>
>> Well, I think in the long term we are likely to want the master to
>> have some kind of ability to track the positions of its slaves, even
>> when they are disconnected. And, optionally, to retain the WAL that
>> they need, again even when they are disconnected. If such an ability
>> materializes, this will be moot (even as I think that pg_standby is
>> now largely moot, at least for new installations, now that we have
>> standby_mode=on).
>
> I agree. But just as we had pg_standby for quite a while before we got
> standby_mode=on, I believe we should have pg_retainxlog (or something
> like it) until we have something more integrated.

Yep, not disagreeing.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

After reviewing this, it appears to me that this is really just a very
verbose version of

archive_command = 'sleep $initialsleep; while test $(psql -AtX -c "select pg_xlogfile_name(something) < $$%f$$ collate \"C\";") = t; sleep $sleep; done'

I think it might be better to just document this as an example. I don't
quite see the overhead of maintaining another tool justified.

On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> After reviewing this, it appears to me that this is really just a very
> verbose version of
>
> archive_command = 'sleep $initialsleep; while test $(psql -AtX -c "select pg_xlogfile_name(something) < $$%f$$ collate \"C\";") = t; sleep $sleep; done'
>
> I think it might be better to just document this as an example. I don't
> quite see the overhead of maintaining another tool justified.

Well, obviously I don't entirely agree ;)

Yes, it's a convenience command. Like pg_standby was. And like many
other commands that we maintain as part of *core*, such as createuser,
vacuumdb, etc. Those can all be done with an even *simpler* command
than the one you suggest above. So I don't see that as an argument why
it wouldn't be useful.

Also, the command you suggest above does not work on Windows. You can
probably write a .BAT file to do it for you, but I'm pretty sure it's
impossible to do it as an archive_command there.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>> I think it might be better to just document this as an example. I don't
>> quite see the overhead of maintaining another tool justified.

> Well, obviously I don't entirely agree ;)

> Yes, it's a convenience command. Like pg_standby was. And like many
> other commands that we maintain as part of *core*, such as createuser,
> vacuumdb, etc. Those can all be done with an even *simpler* command
> than the one you suggest above. So I don't see that as an argument why
> it wouldn't be useful.

We've discussed removing a lot of those tools, too. Not breaking
backwards compatibility is probably the only reason they're still there.

In the case at hand, I seem to recall from upthread that we expect
this'd be obsolete in a release or two. If that's true then I think
a para or two of documentation is a better idea than a tool we'll be
essentially condemned to keep maintaining forever.

> Also, the command you suggest above does not work on Windows. You can
> probably write a .BAT file to do it for you, but I'm pretty sure it's
> impossible to do it as an archive_command there.

Perhaps we could whip up such a .BAT file and put it in the docs?

regards, tom lane

On Thu, Jan 24, 2013 at 6:25 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Thu, Jan 24, 2013 at 6:04 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>>> I think it might be better to just document this as an example. I don't
>>> quite see the overhead of maintaining another tool justified.
>
>> Well, obviously I don't entirely agree ;)
>
>> Yes, it's a convenience command. Like pg_standby was. And like many
>> other commands that we maintain as part of *core*, such as createuser,
>> vacuumdb, etc. Those can all be done with an even *simpler* command
>> than the one you suggest above. So I don't see that as an argument why
>> it wouldn't be useful.
>
> We've discussed removing a lot of those tools, too. Not breaking
> backwards compatibility is probably the only reason they're still there.
>
> In the case at hand, I seem to recall from upthread that we expect
> this'd be obsolete in a release or two. If that's true then I think
> a para or two of documentation is a better idea than a tool we'll be
> essentially condemned to keep maintaining forever.

Not really sure there is such an expectation - any more than there was
such an expectation when we initially put pg_standby in there. It
would be *possible* to do it, certainly. But it's not like we have an
actual plan. And AFAIK the stuff that was discussed upthread was a
simplified version of it - not the full flexibility.

That said, it's certainly a point that we'd have to maintain it. But I
don't see why we'd have to maintain it beyond the point where we
included the same functionality in core, if we did.

>> Also, the command you suggest above does not work on Windows. You can
>> probably write a .BAT file to do it for you, but I'm pretty sure it's
>> impossible to do it as an archive_command there.
>
> Perhaps we could whip up such a .BAT file and put it in the docs?

That would probably work, yes.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Sat, Jan 5, 2013 at 11:11 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
>> On 1/3/13 12:30 PM, Robert Haas wrote:
>>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> Any particular reason? It goes pretty tightly together with
>>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
>>>> But if you have a good argument against it, I can change my mind :)
>>>
>>> Mostly that it seems like a hack, and I suspect we may come up with a
>>> better way to do this in the future.
>>
>> It does seem like a hack. Couldn't this be implemented with a backend
>> switch instead?
>
> It definitely is a bit of a hack.
>
> I assume by backend switch you mean guc, right? If so, no, not easily
> so. Because it's the archiver process that does the deleting.

The process which deletes the old WAL files is the checkpointer. The
checkpointer can access to the shared memory and know the location
of the WAL record which has been already replicated to the standby.
ISTM it's not difficult to implement the logic which pg_retainxlog provides
into the checkpointer. How about just changing the checkpointer so
that it checks whether the WAL file to delete has been already not
only archived but also replicated if GUC flag is enabled?

Regards,

--
Fujii Masao

On 2013-01-26 02:21:00 +0900, Fujii Masao wrote:
> On Sat, Jan 5, 2013 at 11:11 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> > On Fri, Jan 4, 2013 at 7:13 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> >> On 1/3/13 12:30 PM, Robert Haas wrote:
> >>> On Thu, Jan 3, 2013 at 11:32 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> >>>> Any particular reason? It goes pretty tightly together with
> >>>> pg_receivexlog, which is why I'd prefer putting it alongside that one.
> >>>> But if you have a good argument against it, I can change my mind :)
> >>>
> >>> Mostly that it seems like a hack, and I suspect we may come up with a
> >>> better way to do this in the future.
> >>
> >> It does seem like a hack. Couldn't this be implemented with a backend
> >> switch instead?
> >
> > It definitely is a bit of a hack.
> >
> > I assume by backend switch you mean guc, right? If so, no, not easily
> > so. Because it's the archiver process that does the deleting.
>
> The process which deletes the old WAL files is the checkpointer. The
> checkpointer can access to the shared memory and know the location
> of the WAL record which has been already replicated to the standby.
> ISTM it's not difficult to implement the logic which pg_retainxlog provides
> into the checkpointer. How about just changing the checkpointer so
> that it checks whether the WAL file to delete has been already not
> only archived but also replicated if GUC flag is enabled?

The problem with that is that to implement it robustly we would need
persistent state about the replicas.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 1/25/13 12:24 PM, Andres Freund wrote:
> On 2013-01-26 02:21:00 +0900, Fujii Masao wrote:
>> The process which deletes the old WAL files is the checkpointer. The
>> checkpointer can access to the shared memory and know the location
>> of the WAL record which has been already replicated to the standby.
>> ISTM it's not difficult to implement the logic which pg_retainxlog provides
>> into the checkpointer. How about just changing the checkpointer so
>> that it checks whether the WAL file to delete has been already not
>> only archived but also replicated if GUC flag is enabled?

That makes sense.

> The problem with that is that to implement it robustly we would need
> persistent state about the replicas.

Well, pg_retainxlog kind of handwaves around that. If you use it in the
default mode, it assumes that the pg_receivexlog entries in
pg_stat_replication are that state. And then it says, if you use other
kinds of clients, you need to keep track of that state yourself. But
that seems to assume that pg_receivexlog never disconnects (thus losing
its entries from pg_stat_replication). (pg_receivexlog is designed to
automatically reconnect on connection loss, so this possibility was
obviously thought about.)

So it seems to me this just doesn't work (this way).