Missing docs for SR

Hackers,

So, here's a must-fix item for SR for release: we need adequate docs.
I'm happy to write these but *I* need to understand the answers first.

The current docs and wiki page do not explain:

* How (technically) the slave listens for LSNs
* Does the walreceiver need the archive (via archive_command) copies of
the WAL files after it's caught up with the master?
** If not, then what are we supposed to do with them? Write a cron job
to delete them every minute?
** If so, can we somehow use pg_standby with SR?

I've tried to dig this information out of the wiki and mailing list
archives and can't quite figure it out. Is there a tech doc which was
not posted anywhere public, or do I need to just RTFC?

Also, given that recovery.conf has become a key part of our replication,
shouldn't we supply a recovery.example file which people can rename and
edit? Happy to write this too.

--Josh Berkus

On Wed, 2010-01-20 at 11:30 +1300, Josh Berkus wrote:
>
> Also, given that recovery.conf has become a key part of our
> replication, shouldn't we supply a recovery.example file which people
> can rename and edit? Happy to write this too.

We ship it already:

src/backend/access/transam/recovery.conf.sample

and it has sections for HS and SR.
--
Devrim GÜNDÜZ, RHCE
Command Prompt - http://www.CommandPrompt.com
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

On 1/20/10 12:00 PM, Devrim GÜNDÜZ wrote:
> On Wed, 2010-01-20 at 11:30 +1300, Josh Berkus wrote:
>> Also, given that recovery.conf has become a key part of our
>> replication, shouldn't we supply a recovery.example file which people
>> can rename and edit? Happy to write this too.
>
> We ship it already:
>
> src/backend/access/transam/recovery.conf.sample
>
> and it has sections for HS and SR.

Well, it doesn't get created in PGDATA, and the docs don't mention the
above location that I could find.

Can we get initdb to copy it to whereever the config location is? That
seems a lot more administrator-friendly than asking people to look in
the source code.

--Josh Berkus

On Wed, 2010-01-20 at 13:36 +1300, Josh Berkus wrote:

> Well, it doesn't get created in PGDATA, and the docs don't mention the
> above location that I could find.

It is installed under share/ directory for source installations. RPMs
install it under /usr/share/pgsql.

> Can we get initdb to copy it to whereever the config location is?
> That seems a lot more administrator-friendly than asking people to
> look in the source code.

Even though I agree that this file became really important for us, I
think we don't copy any .sample files under $PGDATA.

Maybe we should clarify docs and point admins to relevant dirs.
--
Devrim GÜNDÜZ, RHCE
Command Prompt - http://www.CommandPrompt.com
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org Twitter: http://twitter.com/devrimgunduz

Josh Berkus wrote:
> On 1/20/10 12:00 PM, Devrim GÜNDÜZ wrote:
>
>> On Wed, 2010-01-20 at 11:30 +1300, Josh Berkus wrote:
>>
>>> Also, given that recovery.conf has become a key part of our
>>> replication, shouldn't we supply a recovery.example file which people
>>> can rename and edit? Happy to write this too.
>>>
>> We ship it already:
>>
>> src/backend/access/transam/recovery.conf.sample
>>
>> and it has sections for HS and SR.
>>
>
> Well, it doesn't get created in PGDATA, and the docs don't mention the
> above location that I could find.
>
> Can we get initdb to copy it to whereever the config location is? That
> seems a lot more administrator-friendly than asking people to look in
> the source code.
>
>
>

Devrim gave you the wrong location. It is already installed where we put
other samples like postgresql.conf.sample, namely $SHAREDIR.

Nobody has to look in the source.

I don't think initdb should copy it by default, many users won't want
such an animal at all. Maybe an initdb switch would meet the case.

cheers

andrew

> Devrim gave you the wrong location. It is already installed where we put
> other samples like postgresql.conf.sample, namely $SHAREDIR.
>
> Nobody has to look in the source.
>
> I don't think initdb should copy it by default, many users won't want
> such an animal at all. Maybe an initdb switch would meet the case.

OK, then it just needs to be in the docs.

So my other questions about the recovery files remain ...

--Josh

Josh Berkus wrote:
> So, here's a must-fix item for SR for release: we need adequate docs.
> I'm happy to write these but *I* need to understand the answers first.

Many thanks!

> The current docs and wiki page do not explain:
>
> * How (technically) the slave listens for LSNs

Hmm, not sure what you mean. In a nutshell, the slave connects to the
master over a libpq connection, and master sends the logs as they are
generated.

> * Does the walreceiver need the archive (via archive_command) copies of
> the WAL files after it's caught up with the master?

No. But if the the connection is lost and standby falls behind enough
that the logs are deleted in the master already, it will need the
archive to catch up again.

> ** If so, can we somehow use pg_standby with SR?

No. pg_standby would do the waiting, and the streaming would never start.

I've been wondering if we should actually deprecate pg_standby in favor
of having the startup process do the retrying. So you would configure
restore_command to use 'cp' as in normal archive recovery, and the
server would retry running it every few seconds. It seems we're going to
need that retry logic in the slave anyway, to catch up automatically
after a lost connection. Once it's in there, you could use it without
streaming replication just as well.

> I've tried to dig this information out of the wiki and mailing list
> archives and can't quite figure it out. Is there a tech doc which was
> not posted anywhere public, or do I need to just RTFC?

Feel free to ask.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, Jan 20, 2010 at 7:30 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> So, here's a must-fix item for SR for release: we need adequate docs.
> I'm happy to write these but *I* need to understand the answers first.

Thanks a lot!

> The current docs and wiki page do not explain:
>
> * How (technically) the slave listens for LSNs

(Though I might not have understood your point correctly,) LSN is sent
from the master together with the WAL records. The protocol which SR
uses has been documented in the following page.
http://developer.postgresql.org/pgdocs/postgres/protocol-replication.html

> * Does the walreceiver need the archive (via archive_command) copies of
> the WAL files after it's caught up with the master?

No. And, an archived WAL file is not required for the slave even before
it's caught up with the master.

When the slave is started from the base backup;

1. The startup process tries to perform a normal archive recovery. If
restore_command is not supplied in the recovery.conf, only the WAL
files in pg_xlog are replayed. So restore_command is optional for SR.

2. When the startup process finds the invalid record (including "ENOENT"
of the next WAL file), it requests the postmaster to start walreceiver
process.

3. The walreceiver connects to the master and requests the WAL following
the LSN of that invalid record. Then the WAL records are shipped
continuously to the walreceiver, and written to the slave's disk.

OTOH, the startup process waits until the next record has been written
by the walreceiver, and then reads and applies it. The startup process
continues this stop-and-go recovery.

When you use the old base backup for the slave, not all of the WAL files
required for the slave exist in the master's pg_xlog, i.e., the master
might be unable to ship some of those files. In this case, if you use the
restore_command which accesses the master's archive, those missing files
can be applied on the slave in the phase #1.

> I've tried to dig this information out of the wiki and mailing list
> archives and can't quite figure it out.  Is there a tech doc which was
> not posted anywhere public, or do I need to just RTFC?

Nope. If you have any questions, please feel free to get back to me.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center