register/unregister standby Re: Synchronous replication

On Tue, Aug 10, 2010 at 5:58 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> On 05/08/10 17:14, Fujii Masao wrote:
>>
>> I'm thinking to make users register and unregister each standbys via SQL
>> functions like register_standby() and unregister_standby():
>
> The register/unregister facility should be accessible from the streaming
> replication connection, so that you don't need to connect to any particular
> database in addition to the streaming connection.

Probably I've not understood your point correctly yet.

I think that the advantage of registering standbys is that we can
specify which WAL files the master has to keep for the upcoming
standby. IMO, it's usually called together with pg_start_backup
as follows:

SELECT register_standby('foo', pg_start_backup())

This requests the master keep to all the WAL files following the
backup starting location which pg_start_backup returns. Now we
can do that by using wal_keep_segments, but it's not easy to set
because it's difficult to predict how many WAL files the standby
will require.

So I've thought that the register/unregister facility should be
used from the normal client connection. Why do you think it should
be accessible from the SR connection?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On 30/08/10 15:14, Fujii Masao wrote:
> I think that the advantage of registering standbys is that we can
> specify which WAL files the master has to keep for the upcoming
> standby. IMO, it's usually called together with pg_start_backup
> as follows:
>
> SELECT register_standby('foo', pg_start_backup())
>
> This requests the master keep to all the WAL files following the
> backup starting location which pg_start_backup returns.

Hmm, that's clever. I was thinking that you'd initialize the standby
from an existing backup, and in that context the standby would not need
to connect to the master except via the replication connection. To take
a base backup, you'll need not only that but also access to the
filesystem in the master, ie. shell access.

There's been some talk of being able to stream a base backup over the
replication connection too, which would be extremely handy. And that
makes my point even stronger that registering a standby should be
possible via the replication connection.

Of course, you could well expose the functionality as both a built-in
function and a command in replication mode, so this detail isn't really
that important right now.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> Hmm, that's clever. I was thinking that you'd initialize the standby from an
> existing backup, and in that context the standby would not need to connect
> to the master except via the replication connection. To take a base backup,
> you'll need not only that but also access to the filesystem in the master,
> ie. shell access.

In fact you don't need shell access here, it's rather easy to stream the
base backup from the libpq connection, as implemented here :

http://github.com/dimitri/pg_basebackup

> There's been some talk of being able to stream a base backup over the
> replication connection too, which would be extremely handy.

Yes please ! :)
--
dim

On Thu, Sep 2, 2010 at 6:41 PM, Dimitri Fontaine <dfontaine(at)hi-media(dot)com> wrote:
> In fact you don't need shell access here, it's rather easy to stream the
> base backup from the libpq connection, as implemented here :
>
>  http://github.com/dimitri/pg_basebackup
>
>> There's been some talk of being able to stream a base backup over the
>> replication connection too, which would be extremely handy.
>
> Yes please ! :)

One issue of the base backup function is that the operation will
be a long transaction. So, non-transactional special commands,
as like as VACUUM, would be better in terms of performance.
For example, CREATE or ALTER REPLICATION.

Of course, function-based approach is more flexible and
less invasive to the SQL parser. There are trade-offs.

--
Itagaki Takahiro

On 30 August 2010 13:14, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> I think that the advantage of registering standbys is that we can
> specify which WAL files the master has to keep for the upcoming
> standby. IMO, it's usually called together with pg_start_backup
> as follows:
>
>    SELECT register_standby('foo', pg_start_backup())
>
> This requests the master keep to all the WAL files following the
> backup starting location which pg_start_backup returns. Now we
> can do that by using wal_keep_segments, but it's not easy to set
> because it's difficult to predict how many WAL files the standby
> will require.

+1 I don't like the idea of having to guess how many WAL files you
think you'll need to keep around.

And if these standby instances have to register, could there be a view
to list subscriber information?

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935

Itagaki Takahiro <itagaki(dot)takahiro(at)gmail(dot)com> writes:
>>  http://github.com/dimitri/pg_basebackup
>>
>>> There's been some talk of being able to stream a base backup over the
>>> replication connection too, which would be extremely handy.
>>
>> Yes please ! :)
>
> One issue of the base backup function is that the operation will
> be a long transaction. So, non-transactional special commands,
> as like as VACUUM, would be better in terms of performance.
> For example, CREATE or ALTER REPLICATION.

Well, you still need to stream the data to the client in a format it
will understand. Would that be the plan of your command proposal?

> Of course, function-based approach is more flexible and
> less invasive to the SQL parser. There are trade-offs.

Well that was easier for a proof-of-concept at least.
--
Dimitri Fontaine
PostgreSQL DBA, Architecte

On Thu, Sep 2, 2010 at 7:54 PM, Dimitri Fontaine <dfontaine(at)hi-media(dot)com> wrote:
>> One issue of the base backup function is that the operation will
>> be a long transaction. So, non-transactional special commands,
>> as like as VACUUM, would be better in terms of performance.
>> For example, CREATE or ALTER REPLICATION.
>
> Well, you still need to stream the data to the client in a format it
> will understand.

True, but using libpq connection might be not the most important thing.
The most simplest proof-of-concept might be system("rsync") in the function ;-)

> Would that be the plan of your command proposal?

What I meant was function-based maintenance does not work well in some
cases. I heard before pg_start_backup( no-fast-checkpoint ) caused table
bloating problem because it was a long transaction for 20+ minutes.
The backup function would have the similar issue.

--
Itagaki Takahiro