Re: formula about the number of WAL files

Hi,

----------------
29.4. WAL Configuration

There will always be at least one WAL segment file, and will normally not
be more than (2 + checkpoint_completion_target) * checkpoint_segments + 1
or checkpoint_segments + wal_keep_segments + 1 files.
----------------

The above formula is wrong. The correct is

(2 + checkpoint_completion_target) * checkpoint_segments +
wal_keep_segments + 1

The attached patch fixes this fault.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, 2010-10-14 at 12:00 +0900, Fujii Masao wrote:
> ----------------
> 29.4. WAL Configuration
>
> There will always be at least one WAL segment file, and will normally not
> be more than (2 + checkpoint_completion_target) * checkpoint_segments + 1
> or checkpoint_segments + wal_keep_segments + 1 files.
> ----------------
>
> The above formula is wrong. The correct is
>
> (2 + checkpoint_completion_target) * checkpoint_segments +
> wal_keep_segments + 1
>
> The attached patch fixes this fault.

The current docs are wrong, but your re-definition doesn't match the
actual code in CreateCheckPoint().

I've updated the docs to reflect how the code behaves.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Training and Services

On Fri, Oct 15, 2010 at 6:19 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Thu, 2010-10-14 at 12:00 +0900, Fujii Masao wrote:
>> ----------------
>> 29.4. WAL Configuration
>>
>> There will always be at least one WAL segment file, and will normally not
>> be more than (2 + checkpoint_completion_target) * checkpoint_segments + 1
>> or checkpoint_segments + wal_keep_segments + 1 files.
>> ----------------
>>
>> The above formula is wrong. The correct is
>>
>>     (2 + checkpoint_completion_target) * checkpoint_segments +
>> wal_keep_segments + 1
>>
>> The attached patch fixes this fault.
>
> The current docs are wrong, but your re-definition doesn't match the
> actual code in CreateCheckPoint().
>
> I've updated the docs to reflect how the code behaves.

Hmm.. I seem to be very mixed up. Let me consider the formula again.

Basically checkpoint deletes the WAL files which are older than
the prior checkpoint's redo location. So at least WAL files which
were generated from the prior ckpt start to current ckpt end can
exist in pg_xlog directory. The number of those WAL files is

(1 + checkpoint_completion_target) * checkpoint_segments

But if this number is smaller than wal_keep_segments, checkpoint
leaves wal_keep_segments WAL files in pg_xlog instead. So, from
this aspect, the maximum number of WAL files in pg_xlog is

(1 + checkpoint_completion_target) * checkpoint_segments

or

wal_keep_segments

After checkpoint deletes old WAL files, the number of WAL files in
pg_xlog continues to increase until subsequent checkpoint will have
deleted WAL files. The increased number is

checkpoint_segments

Furthermore, one WAL file might be preallocated aside from the above.

As the result of the above, the maximum number of WAL files in
pg_xlog is

(2 + checkpoint_completion_target) * checkpoint_segments + 1

or

wal_keep_segments + checkpoint_segments + 1

So the original description seems to be correct. Am I missing
something?

# My previous proposal is definitely wrong. Sorry for noise.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, 2010-10-15 at 19:44 +0900, Fujii Masao wrote:

> As the result of the above, the maximum number of WAL files in
> pg_xlog is
>
> (2 + checkpoint_completion_target) * checkpoint_segments + 1
>
> or
>
> wal_keep_segments + checkpoint_segments + 1
>
> So the original description seems to be correct. Am I missing
> something?

After some thought, I agree.

> # My previous proposal is definitely wrong. Sorry for noise.

I'd say the definition of wal_keep_segments isn't good, which is what is
causing the problem here. Will sort that out another day.

I will update the current code with some comments, so this doesn't
happen again. Will do this on Tuesday now, no time left here.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Training and Services

On Fri, Oct 15, 2010 at 9:48 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Fri, 2010-10-15 at 19:44 +0900, Fujii Masao wrote:
>
>> As the result of the above, the maximum number of WAL files in
>> pg_xlog is
>>
>>     (2 + checkpoint_completion_target) * checkpoint_segments + 1
>>
>>     or
>>
>>     wal_keep_segments + checkpoint_segments + 1
>>
>> So the original description seems to be correct. Am I missing
>> something?
>
> After some thought, I agree.

I've put back the original formula.

>> # My previous proposal is definitely wrong. Sorry for noise.
>
> I'd say the definition of wal_keep_segments isn't good, which is what is
> causing the problem here. Will sort that out another day.
>
> I will update the current code with some comments, so this doesn't
> happen again. Will do this on Tuesday now, no time left here.

This is probably a good idea, because the comments in xlog.c are a bit
sparse on this point, but I haven't tried to do it.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company