Re: Memory leakage associated with plperl spi_prepare/spi_freeplan

I looked into the problem described here:
http://www.postgresql.org/message-id/5125087D.8090105@deriva.de

The core of the problem is that plperl's plperl_spi_prepare() sets up
input conversion functions for the parameters of a prepared query
using perm_fmgr_info(), which allocates FmgrInfo structs for the
I/O functions in TopMemoryContext and makes their fn_mcxt values point
to TopMemoryContext too. So when domains.c allocates assorted stuff
in fn_mcxt, that stuff lives forever ... but guess what, the prepared
query doesn't. So we have a memory leak on every use of spi_freeplan().

The leak is particularly egregious for this specific test case, where
an 8K-or-so ExprContext is made as a consequence of domain_check_input's
call to CreateStandaloneExprContext(); but it would add up to something
noticeable eventually even with input functions as innocuous as int4in.
Proof is to try this:

create or replace function perlleak(n int) returns void as $$
my ($n) = @_;
while ($n--) {
my $stmt = spi_prepare('select $1', 'int');
spi_freeplan($stmt);
}
$$ language plperl volatile strict;

with repeat count of a million or so.

I'm surprised we've not seen this reported before --- maybe people
don't tend to use spi_freeplan() much in plperl.

I'm inclined to think the right fix is to make a small memory context
for each prepared plan made by plperl_spi_prepare(). The qdesc for it
could be made right in the context (getting rid of the unchecked
malloc's near the top of the function), the FmgrInfos and their
subsidiary data could live there too, and plperl_spi_freeplan could
replace its retail free's with a single MemoryContextDelete.

Not being particularly a plperl user, I don't really want to code and
test this. Any takers?

regards, tom lane

On Tue, Feb 26, 2013 at 3:56 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> I'm inclined to think the right fix is to make a small memory context
> for each prepared plan made by plperl_spi_prepare(). The qdesc for it
> could be made right in the context (getting rid of the unchecked
> malloc's near the top of the function), the FmgrInfos and their
> subsidiary data could live there too, and plperl_spi_freeplan could
> replace its retail free's with a single MemoryContextDelete.
>
>
Seemed fairly trivial, find the above approach in the attached. I added a
CHECK_FOR_INTERRUPTS() at the top of plperl_spi_prepare(), it was fairly
annoying that I couldn't ctrl+c my way out of test function.

One annonce is it still leaks :-(. I tracked it down and it seemed to stem
from parseTypeString(). I chased down the rabbit hole for a bit, but
nothing jumped out... raw_parser() is a bit of a black box to me. Adding
the seemingly obvious list_free(raw_parsetree_list); or setting the memory
context before parseTypeString() didn't seem to do much.

It would be nice to squish the other leaks due to perm_fmgr_info()... but
this is a start.

Alex Hunsaker <badalex(at)gmail(dot)com> writes:
> On Tue, Feb 26, 2013 at 3:56 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I'm inclined to think the right fix is to make a small memory context
>> for each prepared plan made by plperl_spi_prepare(). The qdesc for it
>> could be made right in the context (getting rid of the unchecked
>> malloc's near the top of the function), the FmgrInfos and their
>> subsidiary data could live there too, and plperl_spi_freeplan could
>> replace its retail free's with a single MemoryContextDelete.

> Seemed fairly trivial, find the above approach in the attached.

Applied with some fixes.

> I added a
> CHECK_FOR_INTERRUPTS() at the top of plperl_spi_prepare(), it was fairly
> annoying that I couldn't ctrl+c my way out of test function.

Good idea, but it wasn't safe where it was --- needs to be inside the
PG_TRY(), so as to convert from postgres to perl error handling.

> One annonce is it still leaks :-(.

I fixed that, at least for the function-lifespan leakage from
spi_prepare() --- is that what you meant?

> It would be nice to squish the other leaks due to perm_fmgr_info()... but
> this is a start.

Yeah, I'm sure there's more left to do in the area --- but at least you
can create and free plans without it leaking.

regards, tom lane

On Fri, Mar 1, 2013 at 7:38 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Alex Hunsaker <badalex(at)gmail(dot)com> writes:
>
> Applied with some fixes.

Thanks! Your version looks much better than mine.

> > One annonce is it still leaks :-(.
>
> I fixed that, at least for the function-lifespan leakage from
> spi_prepare() --- is that what you meant?
>

Yep, I don't see the leak with your version.