Re: [RFC, POC] Don't require a NBuffer sized PrivateRefCount array of local buffer pins

On 2014-04-09 08:22:15 -0400, Robert Haas wrote:
> On Wed, Apr 9, 2014 at 5:34 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> > We've got to the stage now that saving this much memory is essential,
> > so this patch is a must-have.
> >
> > The patch does all I would expect and no more, so approach and details
> > look good to me.
> >
> > Performance? Discussed many years ago, but I suspect the micro-tuning
> > of those earlier patches wasn't as good as it is here.
>
> I think this approach is practically a slam-dunk when the number of
> pins is small (as it typically is). I'm less clear what happens when
> we overflow from the small array into the hashtable. That certainly
> seems like it could be a loss, but how do we construct such a case to
> test it? A session with lots of suspended queries? Can we generate a
> regression by starting a few suspended queries to use up the array
> elements, and then running a scan that pins and unpins many buffers?

I've tried to reproduce problems around this (when I wrote this), but
it's really hard to construct cases that need more than 8 pins. I've
tested performance for those cases by simply not using the array, and
while the performance suffers a bit, it's not that bad.

> One idea is: if we fill up all the array elements and still need
> another one, evict all the elements to the hash table and then start
> refilling the array. The advantage of that over what's done here is
> that the active scan will always being using an array slot rather than
> repeated hash table manipulations. I guess you'd still have to probe
> the hash table repeatedly, but you'd avoid entering and removing items
> frequently.

We could do that, but my gut feeling is that it's not necessary. There'd
be some heuristic to avoid doing that all the time, otherwise we'd
probably regress.
I think the fact that we pin/unpin very frequently will put frequently
used pins to the array most of the time.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services