pg_archive_bypass

Hi,

I tend to consider it a bug that there's no known way under windows to
use the same trick as under Unix by using '/usr/bin/true' as your
archive command. And this Unix trick itself does feel like a hack.

Also I'd very much like to be able to recommend (even if not change the
official defaults) to setup wal_level to archive, archive_mode=on and
archive_command=pg_archive_bypass, so that the day you have a HA budget
ain't the day you're going to restart the server to enable the fault
tolerance settings…

So please find attached a very simple "let's see about it" patch to
implement an internal archive_command that just returns true and is
called pg_archive_bypass. It's missing documentation, which I'll provide
if needed (meaning there's some will to consider applying such a patch).

Regards,
--
dim

On Mon, 2010-06-14 at 12:39 +0200, Dimitri Fontaine wrote:

> I tend to consider it a bug that there's no known way under windows to
> use the same trick as under Unix by using '/usr/bin/true' as your
> archive command. And this Unix trick itself does feel like a hack.
>
> Also I'd very much like to be able to recommend (even if not change the
> official defaults) to setup wal_level to archive, archive_mode=on and
> archive_command=pg_archive_bypass, so that the day you have a HA budget
> ain't the day you're going to restart the server to enable the fault
> tolerance settings…
>
> So please find attached a very simple "let's see about it" patch to
> implement an internal archive_command that just returns true and is
> called pg_archive_bypass. It's missing documentation, which I'll provide
> if needed (meaning there's some will to consider applying such a patch).

ISTM like a good idea to have a couple of more obvious commands provided
purely as "internal commands". If we do this on restore_command as well,
we can skip pg_archivecleanup completely, for example.

So, not for 9.0.

But I like concept for 9.1. Would need to be coded as some kind of
escape phrase, followed by other command. So not just a quick hack with
this one exception.

Something like
archive_command = 'pg_archive_internal:true'

Wouldn't rush to another patch though, needs agreement first.

--
Simon Riggs www.2ndQuadrant.com

On 14/06/10 13:39, Dimitri Fontaine wrote:
> I tend to consider it a bug that there's no known way under windows to
> use the same trick as under Unix by using '/usr/bin/true' as your
> archive command. And this Unix trick itself does feel like a hack.
>
> Also I'd very much like to be able to recommend (even if not change the
> official defaults) to setup wal_level to archive, archive_mode=on and
> archive_command=pg_archive_bypass, so that the day you have a HA budget
> ain't the day you're going to restart the server to enable the fault
> tolerance settings…

That particular use case is better handled by making archive_mode
changeable on-the-fly. Now that we have wal_level as a separate GUC, it
shouldn't be too hard.

I like internal commands as a solution in general, but it's still in
search of a problem..

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> On 14/06/10 13:39, Dimitri Fontaine wrote:
>> I tend to consider it a bug that there's no known way under windows to
>> use the same trick as under Unix by using '/usr/bin/true' as your
>> archive command. And this Unix trick itself does feel like a hack.
>>
>> Also I'd very much like to be able to recommend (even if not change the
>> official defaults) to setup wal_level to archive, archive_mode=on and
>> archive_command=pg_archive_bypass, so that the day you have a HA budget
>> ain't the day you're going to restart the server to enable the fault
>> tolerance settings…
>
> That particular use case is better handled by making archive_mode changeable
> on-the-fly. Now that we have wal_level as a separate GUC, it shouldn't be
> too hard.

Ok.

> I like internal commands as a solution in general, but it's still in search
> of a problem..

What about /usr/bin/true, or a simple archive where you cp in a given
location (which could happen to be a remote server thanks to unix
network file systems or windows shares), etc. It seems to me those are
existing problem that we solve poorly: let each user face the same
pitfalls (error management).

I also like Simon's approach to have options to internal commands
too. What about this syntax :

guc_name = "pg_internal function_name arg1 %r ... argn"

Then function_name is any SQL callable function and you have the %x
substitution still happening there for you. So you can mix hard-coded
arguments (some path) and dynamic arguments. And you can script your
commands in PL/WhateverU.

It's more involved a patch, of course, but if we do that and provide
some simple commands in -core, then most installations will only use
that and stop bothering writing scripts to handle their simple
cases. Which is what I'm after.

Regards,
--
dim

On Mon, Jun 14, 2010 at 1:55 PM, Dimitri Fontaine
<dfontaine(at)hi-media(dot)com> wrote:
> What about /usr/bin/true, or a simple archive where you cp in a given
> location (which could happen to be a remote server thanks to unix
> network file systems or windows shares), etc. It seems to me those are
> existing problem that we solve poorly: let each user face the same
> pitfalls (error management).
>

I would like to see the case where the archive is just a mounted
directory accessible through the filesystem be handled internally.
Ideally if I could just specify the archive location on both the
master and slave using the same parameter just containing a filesystem
path then I could have the same configuration on both machines except
for the actual parameter which decides whether they're a master or
slave. That would make me much more confident I've configured both
machines properly. All the nodes would have the same information and
they would be deciding for themselves whether to push or pull archives
based on whether they're the master or slave.

--
greg

Dimitri Fontaine wrote:
> Hi,
>
> I tend to consider it a bug that there's no known way under windows to
> use the same trick as under Unix by using '/usr/bin/true' as your
> archive command. And this Unix trick itself does feel like a hack.
>
> Also I'd very much like to be able to recommend (even if not change the
> official defaults) to setup wal_level to archive, archive_mode=on and
> archive_command=pg_archive_bypass, so that the day you have a HA budget
> ain't the day you're going to restart the server to enable the fault
> tolerance settings?
>
> So please find attached a very simple "let's see about it" patch to
> implement an internal archive_command that just returns true and is
> called pg_archive_bypass. It's missing documentation, which I'll provide
> if needed (meaning there's some will to consider applying such a patch).

Turn out 'REM' acts like /bin/true on Windows. I have documented that
fact in the attached, applied patch.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Turn out 'REM' acts like /bin/true on Windows. I have documented that
> fact in the attached, applied patch.

I guess that kills it for the "pg_archive_bypass" internal command, but
in a good way. Thanks!

Regards,
--
Dimitri Fontaine
PostgreSQL DBA, Architecte

Damn, I still baffled that we can escape the internal commands
altogether.