Re: BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)

From: Heikki Linnakangas <hlinnakangas(at)vmware(dot)com>
To: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>, Michael Paquier <michael(dot)paquier(at)gmail(dot)com>
Cc: Jehan-Guillaume de Rorthais <jgdr(at)dalibo(dot)com>, Pg Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)
Date: 2014-10-23 08:09:16
Message-ID: 5448B7AC.4030507@vmware.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On 10/23/2014 08:59 AM, Fujii Masao wrote:
> On Mon, Oct 20, 2014 at 3:26 PM, Michael Paquier
> <michael(dot)paquier(at)gmail(dot)com> wrote:
>> On Fri, Oct 17, 2014 at 10:37 PM, Michael Paquier
>> <michael(dot)paquier(at)gmail(dot)com> wrote:
>>>
>>> On Fri, Oct 17, 2014 at 9:23 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>>>
>>>> In this case, the patch seems to make the restartpoint recycle even WAL files
>>>> which have .ready files and will have to be archived later. Thought?
>>>
>>> The real problem currently is that it is possible to have a segment file not marked as .done during recovery when stream connection is abruptly cut when this segment is switched, marking it as .ready in archive_status and simply letting this segment in pg_xlog because it will neither be recycled nor removed. I have not been able to look much at this code these days, so I am not sure how invasive it would be in back-branches, but perhaps we should try to improve code such as when a segment file is switched and connection to the is cut, we guarantee that this file is completed and marked as .done.
>>
>> I have spent more time on that, with a bit more of underground work...
>> First, the problem can be reproduced most of the time by running this
>> simple command:
>> psql -c 'select pg_switch_xlog()'; pg_ctl restart -m immediate
>
> What about fixing this problem directly? That is, we can make walreceiver
> check whether the end of last received WAL data is the end of current WAL file
> or not, and then close the WAL file and create .done file if the test is true.
>
> This is not a perfect solution. If the standby crashes during very
> short interval
> (i.e., after closing the WAL file and before creating .done file), the problem
> would happen again. But it can really rarely happen, so I don't think that it's
> worth fixing the corner case at least in back-branches. Of course, we can
> find out the "perfect" solution for the master, though.

Sounds reasonable, for back-branches. Although I'm still worried we
might miss some corner-case unless we go with a more wholesale solution.

At least for master, we should consider changing the way the archiving
works so that we only archive WAL that was generated in the same server.
I.e. we should never try to archive WAL files belonging to another timeline.

I just remembered that we discussed a different problem related to this
some time ago, at
http://www.postgresql.org/message-id/20131212.110002.204892575.horiguchi.kyotaro@lab.ntt.co.jp.
The conclusion of that was that at promotion, we should not archive the
last, partial, segment from the old timeline. Also, we could make the
archive recovery more precise in what WAL files it tries to restore. For
example, if we're doing recovery with target timeline 4, but the
timeline history says that segment AB came from timeline 3 (i.e. the
switch to timeline 4 happened within a later segment), we should not try
to restore segment AB from timeline 4. It shouldn't exist, so trying to
restore it is a waste of cycles, but more importantly, if there's some
sort of confusion with the timelines, and the file exists anyway, we
should not restore it. Also, if the timeline history says e.g. that we
switched from TLI 3 to 4 at WAL position 0/AD123456, and we restore
segment AD from timeline 3, we should stop recovering that segment at
position 0/AD123456 and try to find the AD segment from timeline 4. We
currently only do that when streaming, but we should also do that when
restoring from archive.

In summary, let's do something small for back-branches, like what you
suggested. But for master, let's do bigger changes to the timeline
handling. Would you like to write a patch for the back-branches, if I
start working on master?

- Heikki

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Etsuro Fujita 2014-10-23 09:29:07 Incorrect comment in tablecmds.c
Previous Message Fujii Masao 2014-10-23 07:34:07 Re: BUG: *FF WALs under 9.2 (WAS: .ready files appearing on slaves)