Re: MemoryContextAllocHuge(): selectively bypassing MaxAllocSize

On Mon, May 13, 2013 at 7:26 AM, Noah Misch <noah(at)leadboat(dot)com> wrote:

> A memory chunk allocated through the existing palloc.h interfaces is
> limited
> to MaxAllocSize (~1 GiB). This is best for most callers; SET_VARSIZE()
> need
> not check its own 1 GiB limit, and algorithms that grow a buffer by
> doubling
> need not check for overflow. However, a handful of callers are quite
> happy to
> navigate those hazards in exchange for the ability to allocate a larger
> chunk.
>
> This patch introduces MemoryContextAllocHuge() and repalloc_huge() that
> check
> a higher MaxAllocHugeSize limit of SIZE_MAX/2. Chunks don't bother
> recording
> whether they were allocated as huge; one can start with palloc() and then
> repalloc_huge() to grow the value.

Since it doesn't record the size, I assume the non-use as a varlena is
enforced only by coder discipline and not by the system?

! * represented in a varlena header. Callers that never use the
allocation as
! * a varlena can access the higher limit with MemoryContextAllocHuge().
Both
! * limits permit code to assume that it may compute (in size_t math)
twice an
! * allocation's size without overflow.

What is likely to happen if I accidentally let a pointer to huge memory
escape to someone who then passes it to varlena constructor without me
knowing it? (I tried sabotaging the code to make this happen, but I could
not figure out how to). Is there a place we can put an Assert to catch
this mistake under enable-cassert builds?

I have not yet done a detailed code review, but this applies and builds
cleanly, passes make check with and without enable-cassert, it does what it
says (and gives performance improvements when it does kick in), and we want
this. No doc changes should be needed, we probably don't want run an
automatic regression test of the size needed to usefully test this, and as
far as I know there is no infrastructure for "big memory only" tests.

The only danger I can think of is that it could sometimes make some sorts
slower, as using more memory than is necessary can sometimes slow down an
"external" sort (because the heap is then too big for the fastest CPU
cache). If you use more tapes, but not enough more to reduce the number of
passes needed, then you can get a slowdown.

I can't imagine that it would make things worse on average, though, as the
benefit of doing more sorts as quicksorts rather than merge sorts, or doing
mergesort with fewer number of passes, would outweigh sometimes doing a
slower mergesort. If someone has a pathological use pattern for which the
averages don't work out favorably for them, they could probably play with
work_mem to correct the problem. Whereas without the patch, people who
want more memory have no options.

People have mentioned additional things that could be done in this area,
but I don't think that applying this patch will make those things harder,
or back us into a corner. Taking an incremental approach seems suitable.

Cheers,

Jeff