Re: Sync Rep: First Thoughts on Code

Hi,

On Thu, Dec 11, 2008 at 7:09 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> > Recent changes I have requested in the architecture are:
>> > * making archiving optional on primary, so we don't need to send WAL
>> > data *twice*.
>>
>> Agreed. I'm not so much worried about the bandwidth, but it's a lot of
>> extra work from administration point of view. It's very hard to get it
>> right, so that you eliminate windows like the above.
>>
>> As the patch stands, if you turn off archiving in the primary, and the
>> standby ever disconnects, even for only a few seconds, the standby will
>> miss any WAL generated until it reconnects, and without archiving
>> there's no way for the standby to get hold of the missed WAL.
>
> I described earlier that archiving would turn back on again if the
> replication ever failed (with correct synchronisation).
>
> All I've asked for is the ability to turn on and turn back on archiving,
> yes, with synchronisation so its safe.
>
> Personally, I think people will laugh if we tell them we decided to ship
> all the data twice and couldn't see another way. That's the kind of
> thing people give presentations at PGcon about...

OK, I will add such archiving feature. My new design of archiving is as follows.

Primary
----------
I extend archive_mode as follows and make the user be able to choose the
archiving strategy on the primary.

- always
The primary always archives the WAL. This is compatible with current (<=8.3)
archive_mode = on.

- none
The primary always doesn't archive the WAL. This is compatible with current
archive_mode = off.

- standalone
The primary doesn't archive the WAL only during replication. If replication is
not in progress, the primary archives the WAL. That is, the primary switches
the modes whenever replication starts / ends.

[FLS->SLS]
When replication starts, the primary disable archiving *after* the switched
WAL file is archived. WAL streaming doesn't need to wait for disablement
of archiving, so the processing on the primary isn't blocked by starting of
replication. But, both WAL streaming and archiving would be in progress
for a while (until the switched WAL file is archived) after
replication starts.

[SLS->FLS]
When replication starts, the primary restarts archiving immediately. This
also doesn't block the processing on the primary. But, this might cause
loss of some files from an archive if archiving is slow on the standby.
The primary should look for the last archived file (by the standby) from
an archive and restart archiving from the subsequent file? Of course,
the primary cannot archive it if it's already removed on the primary.

Standby
-----------
I would add new option for achiving during recovery into recovery.conf
(recovery_archive_mode). Though this option is similar to archive_mode,
merging them would confuse the user more, I think. Or, I should merge?
And, do you want to configure the archive command only for recovery?
If so, I would add new option to specify the archive command during
recovery (recovery_archive_command).

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center