Re: Streaming base backups

2011/1/7 Magnus Hagander <magnus(at)hagander(dot)net>:
> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> 2011/1/5 Magnus Hagander <magnus(at)hagander(dot)net>:
>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
>>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>>>> * Stefan mentiond it might be useful to put some
>>>>> posix_fadvise(POSIX_FADV_DONTNEED)
>>>>>   in the process that streams all the files out. Seems useful, as long as that
>>>>>   doesn't kick them out of the cache *completely*, for other backends as well.
>>>>>   Do we know if that is the case?
>>>>
>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>>>> not already in SHM?
>>>
>>> I think that's way more complex than we want to go here.
>>>
>>
>> DONTNEED will remove the block from OS buffer everytime.
>
> Then we definitely don't want to use it - because some other backend
> might well want the file. Better leave it up to the standard logic in
> the kernel.

Looking at the patch, it is (very) easy to add the support for that in
basebackup.c
That supposed allowing mincore(), so mmap(), and so probably switch
the fopen() to an open() (or add an open() just for mmap
requirement...)

Let's go ?

>
>> It should not be that hard to implement a snapshot(it needs mincore())
>> and to restore previous state. I don't know how basebackup is
>> performed exactly...so perhaps I am wrong.
>
> Uh, it just reads the files out of the filesystem. Just like you'd to
> today, except it's now integrated and streams the data across a
> regular libpq connection.
>
> --
>  Magnus Hagander
>  Me: http://www.hagander.net/
>  Work: http://www.redpill-linpro.com/
>

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support