Re: dynamically allocating chunks from shared memory

On Mon, Jul 26, 2010 at 1:50 PM, Markus Wanner <markus(at)bluegap(dot)ch> wrote:
> Note however, that a thread based design doesn't have this problem *at all*.
> Memory generally is shared (between threads) and you can dynamically
> allocate more or less (until Linux' OOM killer hits you.. yet another
> story). The OS reuses memory you don't currently need even for other
> applications.
>
> Users as well as developers know the threaded model (arguably, much better
> than the process based one). So that's what we get compared to. And what
> developers (including me) are used to.

I'm sort of used to the process model, myself, but I may be in the minority.

> I think we are getting by with fixed allocations at the moment, because we
> did a lot to get by with it. By working around these limitations.
>
> However, that's just my thinking. Thank you for your inputs.

I completely agree with you that fixed allocations suck. We're just
disagreeing (hopefully, in a friendly and collegial fashion) about
what to do about it.

I actually think that memory management is one of the weakest elements
of our current architecture, though I think for somewhat different
reasons than what you're thinking about. Besides the fact that we
have various smaller pools of dynamically shared memory (e.g. a
separate ring of buffers for each SLRU), I'm also unhappy about some
of the things we do with backend-private memory, work_mem being the
biggest culprit by far, because it's very difficult for the DBA to set
the knobs in a way that uses all of the memory he wants to allocate to
the database efficiently no overruns and none left over. The case
where you can count on the database and all of your temporary files,
etc. to fit in RAM is really an exceptional case: in general, you need
to assume that there will be more demand for memory than there will be
memory available, and as much as possible you want the system (rather
than the user) to decide how it should optimally be allocated. The
query planner and executor actually do have most of what is needed to
execute queries using more or less memory, but they lack the global
intelligence needed for intelligent decision-making. Letting the OS
buffer cache rather than the PG buffer cache handle most of the
system's memory helps, but it's not a complete solution.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company