Re: Include WAL in base backup

On Tue, Jan 25, 2011 at 8:21 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> +               elog(DEBUG1, "Going to send wal from %i.%i to %i.%i",
>> +                        logid, logseg,
>> +                        endlogid, endlogseg);
>>
>> %u should be used instead of %i. Or how about logging the filename?
>
> I actually plan to remove that DEBUG output completely (sorry, forgot
> to mention that), I'm not sure it's useful to have it around once we
> apply. Or do you think it would be useful to keep?

Nope. I'm OK to remove that.

>> +                       XLogFileName(xlogname, ThisTimeLineID, logid, logseg);
>> +                       sprintf(fn, "./pg_xlog/%s", xlogname);
>>
>> How about using XLogFilePath? instead?
>
> Can't do that, because we need the "./" part when we call sendFile() -
> it's structured around having that one, since we get it from the file
> name looping.

Understood.

>> +                       if (logid > endptr.xlogid ||
>> +                               (logid == endptr.xlogid && logseg >= endptr.xrecoff / XLogSegSize))
>>
>> Why don't you use endlogseg?
>
> Honestly? Because I thought I added endlogseg just for the debugging
> output, didn't realize I had it down here.
>
> But if I do, then I should not use the XLByteToPrevSeg() per your
> suggestion above, right? Keep it the way it is.

Yes. If we check "logseg >= endlogseg", we should use XLByteToSeg.

>> So, what about sparating the progress report into two parts: one for $PGDATA and
>> tablespaces, another for WAL files?
>
> We can't really do that. We need to send the progress report before we
> begin the tar file, and we don't know how many xlog segments we will
> have at that time. And we don't want to send pg_xlog as a separate tar
> stream, because if we do we won't be able to get the single-tar-output
> in the simple case - thus no piping etc. I actually had the xlog data
> as it's own tar stream in the beginning, but Heikki convinced me of
> the advantage of keeping it in the main one...
>
> What we could do, I guess, is to code pg_basebackup to detect when it
> starts receiving xlog files and then stop increasing the percentage at
> that point, instead just doing a counter?

Umm.. not progressing the report during receiving WAL files seems
also odd. But I don't have another good idea... For now, I'm OK if the
behavior of the progress report is well documented.

> (the discussed changse above have been applied and pushed to github)

Thanks! I'll test and review that.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center