Re: Streaming replication and WAL archive interactions

On 04/21/2015 09:53 AM, Michael Paquier wrote:
> On Thu, Apr 16, 2015 at 8:57 PM, Heikki Linnakangas wrote:
>> Oh, hang on, that's not necessarily true. On promotion, the standby
> archives
>> the last, partial WAL segment from the old timeline. That's just wrong
>> (http://www.postgresql.org/message-id/52FCD37C.3070806@vmware.com), and in
>> fact I somehow thought I changed that already, but apparently not. So
> let's
>> stop doing that.
>
> Er. Are you planning to prevent the standby from archiving the last partial
> segment from the old timeline at promotion?

Yes.

> I thought from previous discussions that we should do it as master
> (be it crashed, burned, burried or dead) may not have the occasion to
> do it. By preventing its archiving you close the door to the case
> where master did not have the occasion to archive it.

The current situation is a mess:

1. Even though we archive the last segment in the standby, there is no
guarantee that the master had archived all the previous segments already.

2. If the master is not totally dead, it might try to archive the same
file with more WAL in it, at the same time or just afterwards, or even
just before the standby has completed promotion. Which copy do you keep
in the archive? Having to deal with that makes the archive_command more
complicated.

Note that even though we don't archive the partial last segment on the
previous timeline, the same WAL is copied to the first segment on the
new timeline. So the WAL isn't lost.

> People may be surprised that a base backup taken from a node that has
> archive_mode = on set (that's the case in a very large number of cases)
> will not be able to work as-is as node startup will fail as follows:
> FATAL: archive_mode='on' cannot be used in archive recovery
> HINT: Use 'shared' or 'always' mode instead.

Hmm, good point.

> One idea would be to simply ignore the fact that archive_mode = on on nodes
> in recovery instead of dropping an error. Note that I like the fact that it
> drops an error as that's clear, I just point the fact that people may be
> surprised that base backups are not working anymore now in this case.

By "ignore", what behaviour do you mean? Would "on" be equivalent to
"shared", "always", or something else?

Or we could keep the current behaviour with archive_mode=on (except for
the last segment thing, which is just wrong), where the standby only
archives the new timeline, and nothing from the previous timelines. Are
the use cases where you'd want that, rather than the new "shared" mode?
I wanted to keep the 'on' mode for backwards-compatibility, but if that
causes more problems, it might be better to just remove it and force the
admin to choose what kind of a setup he has, with "shared" or "always".

> Creating a dependency between the pgstat machinery and the WAL sender looks
> weak to me. For example with this patch a master cannot stop, as it waits
> indefinitely:
> LOG: using stale statistics instead of current ones because stats
> collector is not responding
> LOG: sending archival report:

Hmm, yeah, having walsender to wait for the stats file to appear is not
good.

> You could scan archive_status/ but that would be costly if there are many
> entries to scan and I think that walsender should be highly responsive. Or
> you could directly store the name of the lastly archived WAL segment marked
> as .done in let's say archive_status/last_archived. An entry for that in
> the control file does not seem the right place as a node may not have
> archive_mode enabled that's why I am not mentioning it.

The ways that the archiver process can communicate with the rest of the
system are limited, for the sake of robustness. Writing to the control
file is definitely not OK. I think using the stats collector is OK for
this, but we'll have to arrange it so that the walsender doesn't block
on it, and should probably not force new stat file so often. A 5-10
seconds old stats file would be perfectly fine for this purpose.

- Heikki