Allow substitute allocators for PGresult.

From: Kyotaro HORIGUCHI <horiguchi(dot)kyotaro(at)oss(dot)ntt(dot)co(dot)jp>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Allow substitute allocators for PGresult.
Date: 2011-11-11 09:18:43
Message-ID: 20111111.181843.88228004.horiguchi.kyotaro@oss.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hello. This message is a proposal of a pair of patches that
enables the memory allocator for PGresult in libpq to be
replaced.

The comment at the the begging of pqexpbuffer.c says that libpq
should not rely on palloc(). Besides, Tom Lane said that palloc
should not be visible outside the backend(*1) and I agree with
it.

*1: http://archives.postgresql.org/pgsql-hackers/1999-02/msg00364.php

On the other hand, in dblink, dblink-plus (our product!), and
maybe FDW's connect to other PostgreSQL servers are seem to copy
the result of the query contained in PGresult into tuple store. I
guess that this is in order to avoid memory leakage on
termination in halfway.

But it is rather expensive to copy whole PGresult, and the
significance grows as the data received gets larger. Furthermore,
it requires about twice as much memory as the net size of the
data. And it is fruitless to copy'n modify libpq or reinvent it
from scratch. So we shall be happy to be able to use palloc's in
libpq at least for PGresult for such case in spite of the policy.

For these reasons, I propose to make allocators for PGresult
replaceable.

The modifications are made up into two patches.

1. dupEvents() and pqAddTuple() get new memory block by malloc
currently, but the aquired memory block is linked into
PGresult finally. So I think it is preferable to use
pqResultAlloc() or its descendents in consistensy with the
nature of the place to link.

But there is not PQresultRealloc() and it will be costly, so
pqAddTuple() is not modified in this patch.

2. Define three function pointers
PQpgresult_(malloc|realloc|free) and replace the calls to
malloc/realloc/free in the four functions below with these
pointers.

PQmakeEmptyPGresult()
pqResultAlloc()
PQclear()
pqAddTuple()

This patches make the tools run in backend process and use libpq
possible to handle PGresult as it is with no copy, no more memory.

(Of cource, someone wants to use his/her custom allocator for
PGresult on standalone tools could do that using this feature.)

Three files are attached to this message.

First, the patch with respect to "1" above.
Second, the patch with respect to "2" above.
Third, a very simple sample program.

I have built and briefly tested on CentOS6, with the sample
program mentioned above and valgrind, but not on Windows.

How do you think about this?

Regards,

--
Kyotaro Horiguchi
NTT Open Source Software Center

Attachment Content-Type Size
libpq_dupevents_alloc_r1.patch text/x-patch 2.5 KB
libpq_replasable_alloc_r1.patch text/x-patch 3.8 KB
testlibpq.c.gz application/octet-stream 1.7 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message YAMAMOTO Takashi 2011-11-11 09:26:53 Re: reduce null bitmap size
Previous Message Nikhil Sontakke 2011-11-11 07:57:32 pg_dump: schema with OID XXXXX does not exist - was Concurrent CREATE TABLE/DROP SCHEMA leaves inconsistent leftovers