From: | Cédric Villemain <cedric(dot)villemain(dot)debian(at)gmail(dot)com> |
---|---|
To: | Garick Hamlin <ghamlin(at)isc(dot)upenn(dot)edu> |
Cc: | Magnus Hagander <magnus(at)hagander(dot)net>, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org> |
Subject: | Re: Streaming base backups |
Date: | 2011-01-11 16:39:20 |
Message-ID: | AANLkTi=Nqwh-ExfVMtY-Y-b2VGB2C074wgBEnaz4Ekvm@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Thread: | |
Lists: | pgsql-hackers |
2011/1/11 Garick Hamlin <ghamlin(at)isc(dot)upenn(dot)edu>:
> On Mon, Jan 10, 2011 at 09:09:28AM -0500, Magnus Hagander wrote:
>> On Sun, Jan 9, 2011 at 23:33, Cédric Villemain
>> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> > 2011/1/7 Magnus Hagander <magnus(at)hagander(dot)net>:
>> >> On Fri, Jan 7, 2011 at 01:47, Cédric Villemain
>> >> <cedric(dot)villemain(dot)debian(at)gmail(dot)com> wrote:
>> >>> 2011/1/5 Magnus Hagander <magnus(at)hagander(dot)net>:
>> >>>> On Wed, Jan 5, 2011 at 22:58, Dimitri Fontaine <dimitri(at)2ndquadrant(dot)fr> wrote:
>> >>>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> >>>>>> * Stefan mentiond it might be useful to put some
>> >>>>>> posix_fadvise(POSIX_FADV_DONTNEED)
>> >>>>>> in the process that streams all the files out. Seems useful, as long as that
>> >>>>>> doesn't kick them out of the cache *completely*, for other backends as well.
>> >>>>>> Do we know if that is the case?
>> >>>>>
>> >>>>> Maybe have a look at pgfincore to only tag DONTNEED for blocks that are
>> >>>>> not already in SHM?
>> >>>>
>> >>>> I think that's way more complex than we want to go here.
>> >>>>
>> >>>
>> >>> DONTNEED will remove the block from OS buffer everytime.
>> >>
>> >> Then we definitely don't want to use it - because some other backend
>> >> might well want the file. Better leave it up to the standard logic in
>> >> the kernel.
>> >
>> > Looking at the patch, it is (very) easy to add the support for that in
>> > basebackup.c
>> > That supposed allowing mincore(), so mmap(), and so probably switch
>> > the fopen() to an open() (or add an open() just for mmap
>> > requirement...)
>> >
>> > Let's go ?
>>
>> Per above, I still don't think we *should* do this. We don't want to
>> kick things out of the cache underneath other backends, and since we
>> can't control that. Either way, it shouldn't happen in the beginning,
>> and if it does, should be backed with proper benchmarks.
>
> Another option that occurs to me is an option to use direct IO (or another
> means as needed) to bypass the cache. So rather than kicking it out of
> the cache, we attempt just not to pollute the cache by bypassing it for cold
> pages and use either normal io for 'hot pages', or use a 'read()' to "heat"
> the cache afterward.
AFAIR, even Linus is rejecting the idea to use it seriously, except if
I shuffle in my memory.
>
> Garick
>
>>
>> I've committed the backend side of this, without that. Still working
>> on the client, and on cleaning up Heikki's patch for grammar/parser
>> support.
>>
>> --
>> Magnus Hagander
>> Me: http://www.hagander.net/
>> Work: http://www.redpill-linpro.com/
>>
>> --
>> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-hackers
>
--
Cédric Villemain 2ndQuadrant
http://2ndQuadrant.fr/ PostgreSQL : Expertise, Formation et Support
From | Date | Subject | |
---|---|---|---|
Next Message | Joel Jacobson | 2011-01-11 16:42:23 | Re: pg_depend explained |
Previous Message | Tom Lane | 2011-01-11 16:20:23 | Re: pg_depend explained |