Re: Cascading replication and recovery_target_timeline='latest'

On 03.09.2012 17:40, Heikki Linnakangas wrote:
> On 03.09.2012 16:26, Heikki Linnakangas wrote:
>> On 03.09.2012 16:25, Fujii Masao wrote:
>>> On Tue, Sep 4, 2012 at 7:07 AM, Heikki Linnakangas<hlinnaka(at)iki(dot)fi>
>>> wrote:
>>>> Hmm, I was thinking that when walsender gets the position it can send
>>>> the
>>>> WAL up to, in GetStandbyFlushRecPtr(), it could atomically check the
>>>> current
>>>> recovery timeline. If it has changed, refuse to send the new WAL and
>>>> terminate. That would be a fairly small change, it would just close the
>>>> window between requesting walsenders to terminate and them actually
>>>> terminating.
>>>
>>> Yeah, sounds good. Could you implement the patch? If you don't have
>>> time,
>>> I will....
>>
>> I'll give it a shot..
>
> So, this is what I came up with, please review.

While testing, I bumped into another related bug: When a WAL segment is
restored from the archive, we let a walsender to send that whole WAL
segment to a cascading standby. However, there's no guarantee that the
restored WAL segment is complete. In particular, if a timeline changes
within that segment, e.g 000000010000000000000004, that segment will be
only partially full, and the WAL continues at segment
000000020000000000000004, at the next timeline. This can also happen if
you copy a partial WAL segment to the archive, for example from a
crashed master server. Or if you have set up record-based WAL shipping
not using streaming replication, per
http://www.postgresql.org/docs/devel/static/log-shipping-alternative.html#WARM-STANDBY-RECORD.
That manual page says you can only deal with whole WAL files that way,
but I think with standby_mode='on', that's actually no longer true.

So all in all, it seems like a shaky assumption that once you've
restored a WAL file from the archive, you're free to stream it to a
cascading slave. I think it would be more robust to limit it to
streaming the file only up to the point that it's been replayed - and
thus verified - in the 1st standby. If everyone is OK with that change
in behavior, the fix is simple.

- Heikki